Research map / 3D vision / Face and shape
Towards Variable and Coordinated Holistic Co-Speech Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProbTalk, a VAE-based probabilistic framework with product quantization, a non-autoregressive MaskGIT-style predictor with 2D positional encoding, and a refiner, to generate variable and coordinated holistic (face, hand, body) co-speech motion on the SHOW dataset.
Problem
Generating co-speech holistic 3D avatar motion that is both variable (diverse for similar speech) and coordinated across face, hands, and body, while keeping high-frequency detail and inference efficiency.
Approach
Combines product quantization within a VAE (richer discrete representation with lower quantization error), a MaskGIT-like non-autoregressive predictor whose PQ code embeddings carry 2D (time and subspace) sinusoidal positional encoding, and a deterministic refiner, so that face, hands and body are jointly generated.
Key results
- ProbTalk achieves holistic FGD 3.98 vs 6.60 for TalkShow and 44.60 for Habibie et al. on SHOW.
- Ablation with PQ, MaskGIT and 2D positional encoding and refiner reduces holistic FGD from 6.51 (baseline) to 3.98 (full model).
- MaskGIT with 8 iterations runs at 1067 FPS versus 132 FPS for autoregressive decoding, with a realism trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ProbTalk instead of TalkShow | co-speech holistic motion generation · SHOW test | FGD (Holistic): 6.60 -> 3.98 | Helped |
| use ProbTalk instead of Simultaneous-P | joint holistic motion generation · SHOW test | FGD (Face): 15.49 -> 5.59 | Helped |
| add the Refiner (ablation) | ProbTalk · SHOW test | FGD (Face): 15.49 -> 5.59 | Helped |
| add 2D positional encoding (ablation) | ProbTalk with MaskGIT · SHOW test | FGD (Holistic): 4.89 -> 4.76 | Helped |
| add MaskGIT-like modeling (ablation) | PQ-VAE predictor · SHOW test | FPS: 132 -> 1176 | Helped |
| add MaskGIT-like modeling (ablation) | PQ-VAE predictor · SHOW test | FGD (Holistic): 4.41 -> 4.89 | Hurt |
| add PQ to the VAE (ablation) | motion representation · SHOW test | FGD (Holistic): 6.51 -> 4.41 | Helped |
| add PQ to the VAE (ablation) | motion representation · SHOW test | FPS: 530 -> 132 | Hurt |
| increase codebooks from G=1 to G=4 (K=128) (ablation) | PQ-VAE · SHOW test | FGD (Holistic): 4.73 -> 1.72 | Helped |
| increase MaskGIT iterations from T=1 to T=8 (ablation) | ProbTalk predictor · SHOW test | FGD (Holistic): 4.99 -> 3.98 | Helped |
| increase MaskGIT iterations from T=8 to T=16 (ablation) | ProbTalk predictor · SHOW test | FGD (Holistic): 3.98 -> 3.97 | No clear effect |
| use 2D-Sine positional encoding instead of 1D-Sine (ablation) | ProbTalk predictor · SHOW test | FGD (Holistic): 4.65 -> 3.98 | Helped |
Surprising findings
- Learned 2D positional encoding performed worst among positional encoding variants, while fixed sinusoidal 2D encoding performed best.
- Increasing the PQ codebook size K beyond 128 did not improve reconstruction and hurt FGD, while increasing the number of codebooks G helped substantially.
Limitations
- Evaluated only on the SHOW dataset with 4 speakers.
- MaskGIT speedup comes with reduced realism in body motion (higher FGD body).
Experimental setup
Datasets: SHOW
Benchmarks: SHOW
Metrics: FGD (holistic, body, face) Variance (body, face) FPS L2 reconstruction error (1st and 2nd order)
Baselines: Habibie et al. (2021) whole-body speech-to-motion TalkShow Separate model variant Simultaneous-R Simultaneous-P
Builds on: VQ-VAE MaskGIT TalkShow product quantization VAE
Closest papers by meaning
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.89)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.88)
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.88)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.88)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.87)
- From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.