Research map / Video / Video generation
Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GES-Inter, a 70-hour pseudo-labeled two-speaker concurrent co-speech 3D gesture dataset, and Co3Gesture, a bilateral cooperative diffusion model with a Temporal Interaction Module and mutual attention for coherent two-person gesture generation.
Problem
Generating coherent concurrent co-speech 3D gestures for two interacting speakers in conversation, which existing single-speaker methods and text-to-motion interaction methods do not address; large concurrent gesture datasets are also scarce.
Approach
Two cooperative transformer diffusion branches, each conditioned on its own separated speaker audio plus mixed conversation audio; a Temporal Interaction Module fuses a learned temporal correlation-based interaction weight into each branch, and mutual attention lets branches attend to each other's features.
Key results
- Co3Gesture achieves FGD 0.769, BC 0.692, Diversity 72.824 on GES-Inter, beating InterGen (FGD 1.012, BC 0.670, Diversity 69.455).
- Replacing the bilateral branches with a single branch raises FGD from 0.769 to 1.669.
- User study with 15 volunteers rates the method highest on naturalness (4.4), smoothness (4.5), and interaction coherency (4.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Co³Gesture instead of InterGen | concurrent co-speech gesture generation · GES-Inter | FGD: 1.012 -> 0.769 | Helped |
| use Co³Gesture instead of InterGen | concurrent co-speech gesture generation · GES-Inter | BC: 0.670 -> 0.692 | Helped |
| use Co³Gesture instead of InterGen | concurrent co-speech gesture generation · GES-Inter | Diversity: 69.455$^{\pm1.590}$ -> 72.824$^{\pm2.026}$ | Helped |
| replace TIM with MLP feature fusion (ablation) | Co³Gesture · GES-Inter | FGD: 0.769 -> 1.202 | Hurt |
| remove TIM (ablation) | Co³Gesture · GES-Inter | FGD: 0.769 -> 1.297 | Hurt |
| remove mutual attention (ablation) | Co³Gesture · GES-Inter | FGD: 0.769 -> 0.924 | Hurt |
| remove bilateral diffusion branches (ablation) | Co³Gesture · GES-Inter | FGD: 0.769 -> 1.669 | Hurt |
| remove mixed conversational audio input (ablation) | Co³Gesture · GES-Inter | BC: 0.692 -> 0.656 | Hurt |
| remove separated speaker audio inputs (ablation) | Co³Gesture · GES-Inter | BC: 0.692 -> 0.633 | Hurt |
| remove foot contact loss (ablation) | Co³Gesture · GES-Inter | FGD: 0.769 -> 1.082 | Hurt |
Limitations
- Automatic pose extraction introduces bad instances in the dataset.
- Only upper body is generated; no facial expressions.
- Lacks dedicated interaction metrics for concurrent gesture evaluation.
Experimental setup
Datasets: GES-Inter
Benchmarks: GES-Inter
Metrics: FGD BC Diversity user study naturalness/smoothness/interaction coherency
Baselines: TalkSHOW ProbTalk DiffSHEG EMAGE MDM InterX InterGen
Builds on: DDIM PyMAF-X pyannote-audio WhisperX MDM
Closest papers by meaning
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.91)
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.90)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.90)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.90)
- Towards Variable and Coordinated Holistic Co-Speech Motion Generation (CVPR 2024 · similarity 0.89)
- EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.