Research map / Video / Video generation
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates audio-driven co-speech gesture videos directly by diffusing latent TPS keypoint motion features conditioned on speech, followed by image synthesis and a refinement network.
Problem
Generating co-speech gesture videos (appearance-preserving) rather than skeletons, requiring a motion representation that captures complex body motion and appearance, and temporal alignment with speech for arbitrary-length videos.
Approach
Uses nonlinear TPS-based keypoints as latent motion features, a transformer-based latent motion diffusion model conditioned on WavLM and hand-crafted audio features and source motion, an optimal motion selection module for long sequences at inference, and a UNet-like refinement network focusing on hands, face, and occluded areas.
Key results
- Achieves FGD 18.131 vs 55.655 for ANGIE and 41.626 for MM-Diffusion, and FVD 2058.19 vs 2656.06 for MM-Diffusion.
- User study MOS overall quality 3.77 for the method vs 2.00 for ANGIE and 1.47 for MM-Diffusion (ground truth 4.73).
- Ablation: replacing TPS with MRAA raises FGD from 18.131 to 288.378 and FVD from 2058 to 3035.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TPS-based motion features with MRAA (ablation) | Ours · PATS test set | FGD: 18.131 -> 288.378 | Hurt |
| replace TPS-based motion features with MRAA (ablation) | Ours · PATS test set | FVD: 2058.19 -> 3034.71 | Hurt |
| remove WavLM audio features (ablation) | Ours · PATS test set | FGD: 18.131 -> 37.072 | Hurt |
| remove WavLM audio features (ablation) | Ours · PATS test set | FVD: 2058.19 -> 2053.44 | No clear effect |
| remove refinement network (ablation) | Ours · PATS test set | FVD: 2058.19 -> 2154.00 | Hurt |
| use long noise sampling instead of optimal motion selection (ablation) | Ours · PATS test set | FGD: 18.131 -> 46.055 | Hurt |
| use direct segment concatenation instead of optimal motion selection (ablation) | Ours · PATS test set | FGD: 18.131 -> 20.964 | Hurt |
| use motion-decoupled diffusion model instead of ANGIE | Co-speech gesture video generation · PATS test set | FGD: 55.655 -> 18.131 | Helped |
| use motion-decoupled diffusion model instead of MM-Diffusion | Co-speech gesture video generation · PATS test set | FVD: 2656.06 -> 2058.19 | Helped |
Surprising findings
- ANGIE with motion refinement gets better BAS (0.1504) but worse motion and visual quality, so BAS does not track human-perceived synchrony.
Limitations
- Evaluated on only 4 speakers from the PATS dataset.
- Generated frames still lack fine appearance details compared with ground truth, as noted by user study participants.
- Long-video generation relies on sampling multiple candidates per segment, which adds inference cost.
Experimental setup
Datasets: PATS
Benchmarks: PATS
Metrics: FGD Diversity BAS FVD MOS realness MOS diversity MOS synchrony MOS overall quality
Baselines: ANGIE MM-Diffusion
Builds on: DDPM MRAA ANGIE MM-Diffusion FOMM
Closest papers by meaning
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.91)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.91)
- TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation (ICLR 2025 · similarity 0.90)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.90)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.90)
- AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.