Research map / Video / Video generation
Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IMAE, a two-branch co-speech gesture video generator that entangles implicit SMPL-X pose/expression parameters with audio inside an I2VGen-XL video diffusion model, trained with a two-stage slow-fast schedule.
Problem
Co-speech gesture video generation often relies on strong visual controls (pose images, TPS keypoints), causing blurry hands and distorted fingers, while audio alone is a weak control signal for upper-body gestures.
Approach
Concatenate audio features with noisy SMPL-X pose and expression parameters during training in an auxiliary audio-to-motion branch, then feed the denoised motion representation into the video diffusion model via cross-attention, so inference needs only audio and a reference image; a two-stage slow-fast schedule trains the motion branch on long sequences and the video branch on short clips.
Key results
- Achieves FGD 1.87 and FVD 681.33, versus 3.69 / 816.03 for S2G and 24.24 / 1823.97 for MYA.
- Diversity 273.72 is highest, versus 180.59 for S2G and 224.14 for MYA.
- Removing the first training stage raises FGD to 48.86, the worst among ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the reference encoder (ablation) | IMAE · Constructed dataset test set (four identities) | FVD: 1513.81 -> 681.33 | Helped |
| add motion parameter generation for motion-audio entanglement (ablation) | IMAE · Constructed dataset test set (four identities) | FVD: 2406.91 -> 681.33 | Helped |
| pretrain the audio-to-motion branch in the first training stage (ablation) | IMAE · Constructed dataset test set (four identities) | FGD: 48.86 -> 1.87 | Helped |
| use slow-fast training with longer audio-to-motion sequences (ablation) | IMAE · Constructed dataset test set (four identities) | FVD: 1314.56 -> 681.33 | Helped |
| use IMAE instead of S2G | co-speech gesture video generation · Constructed dataset test set (four identities) | BAS: 0.7280 -> 0.7445 | Helped |
| use IMAE instead of MYA | co-speech gesture video generation · Constructed dataset test set (four identities) | BAS: 0.7452 -> 0.7445 | Hurt |
Surprising findings
- MYA achieves higher BAS (0.7452) than the proposed method (0.7445), which the authors attribute to its audio-to-motion stage capturing gesture-audio peak alignment.
Limitations
- Evaluated only on four identities from one source dataset, with limited identity coverage.
- Lower BAS than MYA on the reported metrics.
- Relies on a custom dataset with no public release stated.
Experimental setup
Datasets: PATS custom four-identity YouTube dataset
Benchmarks: custom four-identity co-speech gesture test set
Metrics: FGD Diversity BAS FVD user study
Baselines: S2G MYA
Builds on: I2VGen-XL DiffSHEG Handiffuser S2G
Closest papers by meaning
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.91)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.90)
- HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation (CVPR 2025 · similarity 0.90)
- TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation (ICLR 2025 · similarity 0.89)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.88)
- EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.