Research map / Vision-language and multimodal / Human and embodied modeling
ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConvoFusion, a time-aware latent diffusion model for monadic and dyadic co-speech gesture synthesis with modality-level and word-level guidance, and releases the DND Group Gesture multi-party dataset.
Problem
Co-speech gesture generation methods capture beat gestures but struggle to produce semantically aligned gestures, lack control over modalities, and existing datasets are mostly monologues or two-person conversations.
Approach
A latent diffusion model over chunked, decoupled body/hand motion latents, with a transformer denoiser conditioned on audio, text, speaker identity and co-participant speech; classifier-free guidance with per-modality null-dropout enables modality-level control, and word-excitation guidance boosts attention to user-selected words for fine-grained semantic gestures.
Key results
- On BEAT, the method reaches FID 271 and beat alignment 0.82, with SRGR 0.365 versus CaMN's 0.443 and FID 142.
- In a user study, monadic results were preferred over CaMN and MLD at roughly 86-91% mean preference on speech alignment and naturalness.
- Word-excitation guidance raised semantic-gesture recall from 0.34 to 0.40 in a gesture-type classifier test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConvoFusion instead of CaMN | Monadic co-speech gesture synthesis · BEAT | BeatAlign: 0.74 -> 0.82 | Helped |
| use ConvoFusion instead of MLD | Monadic co-speech gesture synthesis · BEAT | L1 Div: 5.42 -> 6.24 | Helped |
| use ConvoFusion instead of MLD | Dyadic co-speech gesture synthesis · DND GROUP GESTURE | BeatAlign: 0.96 -> 0.90 | Helped |
| add word-excitation guidance (ablation) | ConvoFusion · BEAT | Semantic gesture recall: 0.34 -> 0.40 | Helped |
| add Laplacian regularization loss (ablation) | ConvoFusion VAE · VAE reconstruction | Smoothness Error: 3.7 × 10−3 -> 3.5 × 10−3 | Helped |
| add Laplacian regularization loss (ablation) | ConvoFusion VAE · VAE reconstruction | Reconstruction Loss: 3 × 10−4 -> 5 × 10−3 | Hurt |
| add time-aware latent representation (ablation) | ConvoFusion VAE · VAE reconstruction | Reconstruction Loss: 9 × 10−3 -> 5 × 10−3 | Helped |
| add time-aware latent representation (ablation) | ConvoFusion · Perpetual gesture generation | Perpetual generation capability | Helped |
| add scale-aware latent representation (ablation) | ConvoFusion VAE · VAE reconstruction | Reconstruction Loss: 5.5 × 10−3 -> 5 × 10−3 | Helped |
| add scale-aware latent representation (ablation) | ConvoFusion VAE · VAE reconstruction | Smoothness Error: 3.7 × 10−3 -> 3.5 × 10−3 | Helped |
Surprising findings
- Multi-Context and DiffGesture achieve the best or competitive beat alignment only through extremely jittery motion, so beat alignment alone is misleading.
- CaMN outperforms the proposed method on FID and SRGR on BEAT, with the proposed method ranking second on those metrics.
Limitations
- Semantic gesture recall remains low (0.40), indicating semantic alignment is still limited.
- Dyadic user-study margins over baselines are significantly smaller than in the monadic setting.
- Baselines that require seed motions were run without them, which may affect comparability.
Experimental setup
Datasets: BEAT DND GROUP GESTURE
Benchmarks: BEAT DND GROUP GESTURE
Metrics: FID Beat Alignment SRGR Diversity L1 Diversity user preference
Baselines: MLD CaMN Multi-Context DiffGesture DiffuGesture
Builds on: MLD classifier-free guidance DDPM CaMN
Closest papers by meaning
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.91)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.90)
- Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis (CVPR 2025 · similarity 0.90)
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.90)
- Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control (AAAI 2024 · similarity 0.89)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.