Research map / Video / Video generation
TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TANGO, a motion-graph gesture video retrieval framework with a learned audio-motion joint embedding (AuMoCLIP) for retrieval and a diffusion-based transition interpolation model (ACInterp) for co-speech body-gesture video generation.
Problem
Generating high-quality co-speech body-gesture videos synchronized with unseen target speech audio from a few-minute single-speaker reference video, addressing audio-motion misalignment and GAN/flow artifacts in prior retrieval-based gesture video reenactment (GVR).
Approach
Replaces GVR's onset/keyword retrieval with a hierarchical (low-level frame-wise and high-level clip-wise) contrastive audio-motion embedding, prunes the motion graph to strongly connected components for infinite-length sampling, and fills transitions with a diffusion interpolator (ACInterp) built on AnimateAnyone that adds homography background offsets and reference-frame conditioning for appeara
Key results
- TANGO achieves FVD 1.379 and FGD 3.714 on Show-Oliver, beating GVR (1.615 / 4.246), though its beat consistency (0.375) trails SpeechDrivenTemplate (0.401).
- AuMoCLIP reaches 65.68% low-level and 19.54% high-level retrieval accuracy versus 35.38% / 1.288% for GVR's onset and keyword matching.
- ACInterp reaches FVD 0.922 and PSNR 35.63 on transition frame generation, beating the best prior method PANR (FVD 1.190).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GVR retrieval and interpolation with TANGO | co-speech video generation · Show-Oliver | FVD: 1.615 -> 1.379 | Helped |
| replace GVR retrieval and interpolation with TANGO | co-speech video generation · Show-Oliver | BC: 0.401 -> 0.375 | Hurt |
| replace GVR retrieval and interpolation with TANGO | co-speech video generation · YouTube Talking Video | FVD: 4.027 -> 3.133 | Helped |
| replace GVR retrieval and interpolation with TANGO | co-speech video generation · YouTube Talking Video | BC: 0.331 -> 0.479 | Helped |
| replace PANR with ACInterp | video interpolation · Show-Oliver interpolation test set | FVD: 1.190 -> 0.922 | Helped |
| use CLS token instead of max pooling (ablation) | AuMoCLIP · AuMoCLIP retrieval | High-level retrieval accuracy: 5.312% -> 11.84% | Helped |
| add time-aligned BERT features (ablation) | AuMoCLIP CLS baseline · AuMoCLIP retrieval | High-level retrieval accuracy: 11.84% to 15.68% | Helped |
| add both Wav2Vec2 and BERT features (ablation) | AuMoCLIP · AuMoCLIP retrieval | High-level retrieval accuracy: 15.68% -> 16.40% | Helped |
| split encoders into low- and high-level features (ablation) | AuMoCLIP · AuMoCLIP retrieval | High-level retrieval accuracy: 16.40% -> 17.83% | Helped |
| train with low-level contrastive learning only (ablation) | AuMoCLIP · AuMoCLIP retrieval | Low-level retrieval accuracy: 65.57% | Helped |
| stop global-loss gradients to the low-level encoder (ablation) | AuMoCLIP · AuMoCLIP retrieval | Low-level retrieval accuracy: 65.57% -> 65.68% | Helped |
| stop global-loss gradients to the low-level encoder (ablation) | AuMoCLIP · AuMoCLIP retrieval | High-level retrieval accuracy: 17.83% -> 19.54% | Helped |
Surprising findings
- Stopping gradients from the high-level contrastive loss into the low-level encoder improved low-level retrieval, and training only the low-level task reached 65.57%, suggesting joint training hurts the low-level encoder.
- Generative audio-to-motion features performed poorly for retrieval (29.03% low-level, 1.403% high-level), barely above keyword/onset baselines.
Limitations
- Linear 2D pose blending assumption holds only for about 83% of clips; non-linear motion between interpolated frames yields differences from ground truth.
- Beat consistency is lower than the fully generative SpeechDrivenTemplate baseline.
- Lip synchronization is handled by separate Wav2Vec2-free post-processing with Wav2Lip, so the pipeline is not end-to-end for face motion.
Experimental setup
Datasets: Show-Oliver (Talkshow) YouTube Video dataset
Benchmarks: Show-Oliver YouTube Talking Video
Metrics: FVD FGD Beat Consistency Diversity PSNR LPIPS MOVIE retrieval accuracy user study preference
Baselines: SpeechDrivenTemplate ANGLE GVR FiLM VFIFormer AnimateAnyone PANR
Builds on: Gesture Video Reenactment AnimateAnyone AnimateDiff Wav2Vec2 MoCoV2
Closest papers by meaning
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.90)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.89)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.88)
- HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation (CVPR 2025 · similarity 0.86)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.86)
- Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.