Research map / Image generation / GANs
DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIDiffGes, a decoupled semi-implicit diffusion framework that uses GAN-style adversarial training plus L2 and auxiliary forward-diffusion losses to generate co-speech gestures in 10 sampling steps.
Problem
Diffusion models produce high-quality co-speech gestures but their many sampling steps make real-time gesture generation impractical.
Approach
Combines large-step diffusion denoising with adversarial marginal matching, an auxiliary forward-diffusion constraint, and separate body/hand decoupling, plus a sequential denoiser that generates root motion conditioned on denoised local body motion.
Key results
- With 10 sampling steps, DIDiffGes reaches FGD 12.57 on ZeroEGGs and 98.6 on BEAT, versus DSG's 15.67 and 113.7 with 1000 steps.
- Inference time is 0.29 ms per frame on ZeroEGGs at 10 steps versus 6.01 ms for DSG, about 15x faster per the paper's claim.
- Compared with other 10-step accelerators, DIDiffGes beats DPM-Solver-2 (FGD 19.92 on ZeroEGGs) and naive DDGAN/SIDDMs (FGD 44.7/49.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sampling steps from 1 to 10 (ablation) | DIDiffGes · ZeroEGGs | FGD: 85.44 -> 12.57 | Helped |
| Increase sampling steps from 10 to 50 (ablation) | DIDiffGes · ZeroEGGs | FGD: 12.57 -> 12.02 | No clear effect |
| Increase sampling steps from 10 to 50 (ablation) | DIDiffGes · ZeroEGGs | Inf. time: 0.29 -> 0.99 | Hurt |
| Add explicit reconstruction loss (weight 1 instead of 0) (ablation) | DIDiffGes · ZeroEGGs | FGD: 93.10 -> 12.74 | Helped |
| Vary reconstruction loss weight from 1 to 10 or 100 (ablation) | DIDiffGes · ZeroEGGs | FGD: 12.74 -> 12.32; 12.74 -> 12.38 | No clear effect |
| Add auxiliary forward diffusion constraint (ablation) | DIDiffGes · ZeroEGGs | FGD: 26.9 -> 12.32 | Helped |
| Use DIDiffGes instead of DSG | gesture generation · ZeroEGG | FGD: 15.67 -> 12.57 | Helped |
| Use DIDiffGes instead of DSG | gesture generation · BEAT | FGD: 113.7 -> 98.6 | Helped |
| Use DIDiffGes instead of DSG | gesture generation · BEAT | BA: 0.89 -> 0.88 | Hurt |
| Use DIDiffGes instead of FreeTalker | gesture generation · ZeroEGG | DIV: 0.84 -> 0.76 | Hurt |
| Use DIDiffGes instead of FreeTalker | gesture generation · BEAT | DIV: 0.77 -> 0.74 | Hurt |
| Use DIDiffGes instead of DPM-Solver-1 (DDIM) at 10 steps | gesture generation · ZeroEGG | FGD: 21.7 -> 12.57 | Helped |
Surprising findings
- Reconstruction loss weight in {1, 10, 100} had little effect on FGD, according to the authors.
- Naive implicit (DDGAN) and semi-implicit (SIDDM) acceleration performed very poorly on gesture generation without the geometric constraints.
Limitations
- Gesture quality relies on explicit geometric constraints; purely implicit or semi-implicit adversarial training alone performs poorly on human motion.
- Comparisons on BEAT required retraining baselines with open-source code, so results depend on those re-implementations.
Experimental setup
Datasets: BEAT ZeroEGGs AIST++
Benchmarks: BEAT ZeroEGGs AIST++
Metrics: FGD BA DIV inference time user study
Baselines: DSG FreeTalker DiffGesture Trimodal HA2G CAMN DPM-Solver DDGAN SIDDMs MLD
Builds on: SIDDM DDGAN MDM DDPM CFG
Closest papers by meaning
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.89)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.88)
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.86)
- TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation (ICLR 2025 · similarity 0.86)
- Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control (AAAI 2024 · similarity 0.86)
- EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.