Research map / Image generation / Autoregressive and latent generation
DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffSHEG, a diffusion transformer that jointly generates speech-driven 3D facial expressions and body gestures with one-way expression-to-gesture information flow, plus outpainting-based (FOPPAS) arbitrary-length real-time sampling.
Problem
Speech-driven co-speech generation has treated facial expressions and gestures separately, and existing holistic methods are deterministic or decouple the two, missing their joint distribution; long sequence generation and real-time inference are also hard for diffusion models.
Approach
A unified diffusion Transformer (UniEG) that feeds the predicted expression x0 from the expression branch (gradient-detached) into the gesture branch, enforcing a uni-directional expression-to-gesture flow, combined with FOPPAS: training-free outpainting (Repaint) with 25-step DDIM for arbitrary-length, streaming generation.
Key results
- On BEAT, DiffSHEG reports FMD 324.67 and FGD 438.93 versus 1055.52 and 1635.44 for CaMN (the strongest prior baseline by FMD).
- On SHOW, DiffSHEG reaches FMD 0.00184 and FED 0.00161 versus TalkSHOW's 0.00219 and 0.00233, and PCM 0.902 versus 0.902 for TalkSHOW... with diversity 0.923 vs 0.740.
- FOPPAS runs at about 31.5 FPS on a single RTX 3090 for one-minute audio (28.6 s), versus 0.44 FPS for DDPM with RePaint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffSHEG instead of LDA | Speech-driven holistic expression and gesture generation · BEAT test | FMD: 688.25 -> 324.67 | Helped |
| Use DiffSHEG instead of TalkSHOW | Speech-driven holistic expression and gesture generation · SHOW test | FMD: 0.00219 -> 0.00184 | Helped |
| Use DiffSHEG instead of TalkSHOW | Speech-driven holistic expression and gesture generation · SHOW test | Div: 0.740 -> 0.923 | Helped |
| Use DiffSHEG instead of LDA | Speech-driven holistic expression and gesture generation · BEAT test | SRGR: 0.215 -> 0.251 | Helped |
| Use 25-step DDIM with FOPPAS instead of default Repaint DDPM sampling | DiffSHEG · 900-frame BEAT sequence | FPS: 0.44 FPS -> 31.5 FPS | Helped |
| Use expression-to-gesture condition flow rather than naïve concatenation (ablation) | DiffSHEG · BEAT test | FMD: 354.60 -> 324.67 | Helped |
| Use expression-to-gesture condition flow rather than naïve concatenation (ablation) | DiffSHEG · SHOW test | FMD: 0.00255 -> 0.00184 | Helped |
| Add gradient cutoff before passing predicted expression to gesture (ablation) | DiffSHEG · BEAT test | FMD: 375.98 -> 324.67 | Helped |
| Add gradient cutoff before passing predicted expression to gesture (ablation) | DiffSHEG · SHOW test | FMD: 0.00259 -> 0.00184 | Helped |
| Add predicted expression conditioning to the gesture branch (ablation) | DiffSHEG · BEAT test | FMD: 369.97 -> 324.67 | Helped |
| Add predicted expression conditioning to the gesture branch (ablation) | DiffSHEG · SHOW test | FMD: 0.00248 -> 0.00184 | Helped |
| Use expression-to-gesture rather than gesture-to-expression condition flow (ablation) | DiffSHEG · BEAT test | FMD: 357.56 -> 324.67 | Helped |
Surprising findings
- A naive concatenation of expression and gesture in one Transformer performed worse than the uni-directional design, and reversing the flow (gesture to expression) hurt expression metrics.
Limitations
- Gesture evaluation on SHOW and BEAT restricted to upper body, despite the framework supporting full-body motion.
- Metrics such as diversity and beat alignment are only references and can be inflated by jittery motion; user study was small (22 subjects).
Experimental setup
Datasets: BEAT SHOW
Benchmarks: BEAT SHOW
Metrics: FMD FED FGD Diversity BA SRGR PCM FPS user preference
Baselines: CaMN DiffGesture DiffuseStyleGesture ListenDenoiseAction TalkSHOW LS3DCG
Builds on: DDPM DDIM RePaint HuBERT Linear self-attention
Closest papers by meaning
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.91)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.90)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.90)
- Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion (CVPR 2024 · similarity 0.89)
- DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech (AAAI 2025 · similarity 0.89)
- Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.