Research map / 3D vision / 3D generation
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMAGE, a masked audio-conditioned transformer that generates full-body co-speech gestures (face, body, hands, global translation) from audio plus optional masked gesture hints, and releases the BEAT2 mesh-level SMPL-X/FLAME dataset.
Problem
Holistic co-speech gesture generation is hard because existing datasets use inconsistent formats (ARKit blendshapes, skeletons, pseudo-GT) and existing models handle only face or body, or cannot accept partial gesture constraints.
Approach
Jointly train masked gesture reconstruction and audio-conditioned gesture generation in one transformer with a switchable cross-attention path, use content-rhythm self-attention to fuse audio features, and decode face and body with four compositional VQ-VAEs plus a global motion predictor, on a mesh-level SMPL-X/FLAME dataset built from BEAT.
Key results
- On BEATv2, EMAGE reaches FGD 5.512 vs 6.209 for TalkSHOW (body realism).
- In a 60-participant user study, EMAGE wins 52.7% holistic preference against TalkSHOW's 34.9%.
- Masked body hints improve FGD on BEATv1.3 from 6.833 to 5.423 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one VQ-VAE for full-body movements (ablation) | EMAGE baseline · BEATv1.3 | FGD: 13.080 -> 9.787 | Helped |
| use four separate VQ-VAEs for face, upper body, hands, and lower body instead of one (ablation) | EMAGE with one VQ-VAE · BEATv1.3 | FGD: 9.787 -> 7.397 | Helped |
| add Content Rhythm Self-Attention (CRA) (ablation) | EMAGE with four VQ-VAEs · BEATv1.3 | FGD: 7.397 -> 6.833 | Helped |
| add masked body gesture hints (ablation) | EMAGE with CRA · BEATv1.3 | FGD: 6.833 -> 5.423 | Helped |
| add Trinity upper-body and audio data during training (ablation) | EMAGE · BEATv1.3 test set | FGD: 5.423 -> 5.319 | Helped |
| add AMASS body and hands data during training (ablation) | EMAGE · BEATv1.3 test set | FGD: 5.423 -> 5.174 | Helped |
Surprising findings
- CodeTalker has higher MSE than EMAGE on BEATv2 yet is subjectively rated more realistic.
Limitations
- Dataset quality comparisons rely on a perceptual study rather than objective measures.
- Baselines that were limited to upper body had to be extended by the authors to produce full-body outputs.
Experimental setup
Datasets: BEAT BEAT2 Trinity AMASS BEATv1.3 BEATv2
Benchmarks: BEATv1.3 BEATv2 BEAT2
Metrics: FGD BC Diversity MSE LVD user preference win rate
Baselines: FaceFormer CodeTalker S2G Trimodal HA2G DisCo CaMN DiffStyleGesture Habibie et al. TalkSHOW
Builds on: FaceFormer TalkSHOW CaMN MoSh++ BEAT
Closest papers by meaning
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.89)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.88)
- Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion (CVPR 2024 · similarity 0.87)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.87)
- Towards Variable and Coordinated Holistic Co-Speech Motion Generation (CVPR 2024 · similarity 0.87)
- TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.