Research map / Video / Video generation
MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MambaTalk, a two-stage co-speech gesture synthesis framework that pairs VQ-VAE discrete motion priors with selective state space (Mamba) models using local and global scan modules, achieving faster inference than diffusion methods.
Problem
Co-speech 3D holistic gesture synthesis is needed in low-latency settings, but RNNs and transformers scale poorly with sequence length and diffusion models are computationally expensive; directly applying Mamba yields jittery gestures and averaged limb motion across body parts.
Approach
Model gestures in discrete VQ-VAE motion codebooks (trained with velocity and acceleration losses, one codebook per body part) and generate codes with selective SSM (Mamba) blocks, using learnable part-specific queries, multi-head attention, and separate local (per-part) and global Mamba scans.
Key results
- On BEAT2, MambaTalk reports FGD 5.366 (x10^-1), BC 7.812, MSE 6.289 (x10^-7) and LVD 6.897 (x10^-5), best on FGD, BC, MSE and LVD versus EMAGE (FGD 5.512).
- Removing VQ-VAE priors raises FGD from 5.366 to 12.051, showing the discrete motion prior is critical.
- Per-second inference takes about 0.019 s total versus 0.644 s for diffusion-based DiffStyleGesture, per the paper's efficiency table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MambaTalk instead of EMAGE | BEAT2 gesture synthesis · BEAT2 test | FGD: 5.512 -> 5.366 | Helped |
| use MambaTalk instead of EMAGE | BEAT2 gesture synthesis · BEAT2 test | Beat Constancy (BC): 7.724 -> 7.812 | Helped |
| use MambaTalk instead of EMAGE | BEAT2 facial motion generation · BEAT2 test | MSE: 7.680 -> 6.289 | Helped |
| use MambaTalk instead of EMAGE | BEAT2 facial motion generation · BEAT2 test | LVD: 7.556 -> 6.897 | Helped |
| use MambaTalk instead of CaMN | BEAT gesture synthesis · BEAT test | FGD: 123.7 -> 51.3 | Helped |
| remove VQVAEs (ablation) | MambaTalk · BEAT2 ablation | FGD: 5.366 -> 12.051 | Hurt |
| remove VQVAEs (ablation) | MambaTalk · BEAT2 ablation | Beat Constancy (BC): 7.812 -> 7.447 | Hurt |
| remove local-scan multi-head cross-attention (ablation) | MambaTalk · BEAT2 ablation | Beat Constancy (BC): 7.812 -> 6.701 | Hurt |
| remove local-scan Mamba (ablation) | MambaTalk · BEAT2 ablation | Diversity: 13.048 -> 12.844 | Hurt |
| remove local-scan Mamba (ablation) | MambaTalk · BEAT2 ablation | FGD: 5.366 -> 7.277 | Hurt |
| remove global-scan multi-head self-attention (ablation) | MambaTalk · BEAT2 ablation | Diversity: 13.048 -> 11.875 | Hurt |
| remove global-scan multi-head self-attention (ablation) | MambaTalk · BEAT2 ablation | FGD: 5.366 -> 6.308 | Helped |
Surprising findings
- Without VQ-VAE priors the user study score collapses to 1.24 average, far below all compared methods.
- Whisper features improve facial metrics while Wav2Vec2 improves body metrics, suggesting encoder choice should differ by body part.
Limitations
- Distinct modules for each body part introduce latency and overhead.
- Diversity across speakers and cultures is not addressed.
- Diversity and BC are not uniformly best; the Mamba-based model is not uniformly superior on all metrics (e.g., Diversity lower than TalkShow).
Experimental setup
Datasets: BEAT2 BEAT
Benchmarks: BEAT2 BEAT
Metrics: FGD Diversity BC MSE LVD SRGR BeatAlign user study ratings (naturalness, appropriateness, synchrony,
Baselines: S2G Trimodal HA2G DisCo CaMN DiffStyleGesture Habible et al. TalkShow EMAGE
Builds on: Mamba VQ-VAE EMAGE HiPPO S4
Closest papers by meaning
- Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control (AAAI 2024 · similarity 0.87)
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.85)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.85)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.85)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.84)
- HumanTOMATO: Text-aligned Whole-body Motion Generation (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.