Research map / Image generation / Autoregressive and latent generation
Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAG-Gesture, an inference-time retrieval-augmented diffusion model that retrieves semantic exemplar gestures from BEAT2 (via LLM gesture-type or discourse-connective retrieval) and injects them via DDIM-inversion latent initialization and retrieval guidance, without training.
Problem
Co-speech gesture generation models produce rhythmic beat gestures but struggle to generate semantically meaningful gestures, which are sparse in datasets.
Approach
Frame semantic gesture synthesis as retrieval-augmented generation: retrieve context-matched exemplar gestures for semantically important words and insert them into a latent diffusion gesture generator at inference time via DDIM inversion latent initialization plus a controllable retrieval guidance term, with no retraining.
Key results
- Trained on all speakers, Ours (w/ LLM & Gesture Type) reaches FID 0.808 vs. 0.570 for EMAGE on the single-speaker setting; on all-speakers evaluation Ours (w/ Discourse) achieves FID 0.447 and BeatAlign 0.471 (ground truth 0.477 FID, 7.29 BeatAlign).
- In user studies, RAG-Gesture (Discourse) is preferred over EMAGE, Audio2Photoreal and ReMoDiffuse for naturalness and appropriateness (e.g., 91.1% and 83.9% vs. EMAGE).
- Inference-time RAG-Gesture outperforms a training-based RAG variant (FID 0.447 vs. 0.525) with similar MPJPE (35.0 vs. 34.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LLM-predicted gesture types for semantic retrieval instead of no retrieval (ablation) | RAG-GESTURE · BEAT2 test, all speakers | BeatAlign: 0.447 -> 0.514 | Helped |
| use discourse-based retrieval instead of no retrieval (ablation) | RAG-GESTURE · BEAT2 test, all speakers | FID: 0.519 -> 0.447 | Helped |
| use LLM gesture-type retrieval rather than discourse retrieval (ablation) | RAG-GESTURE · BEAT2 test, all speakers | naturalness preference: 50.0% vs. 50.0% | No clear effect |
| train with discourse-based local retrievals | ReMoDiffuse · BEAT2 test, all speakers | FID: 1.120 -> 0.525 | Helped |
| use inference-time RAG-GESTURE instead of training-based RAG | gesture generation · BEAT2 test, all speakers | FID: 0.525 -> 0.447 | Helped |
| remove retrieval augmentation (ablation) | RAG-GESTURE · BEAT2 test, all speakers | FID: 0.447 -> 0.519 | Hurt |
| replace latent initialization and retrieval guidance with inpainting (ablation) | RAG-GESTURE · BEAT2 test, all speakers | MPJPE (mm): 35.0 -> 28.1 | Mixed |
| remove retrieval guidance while retaining latent initialization (ablation) | RAG-GESTURE · BEAT2 test, all speakers | FID: 0.447 -> 0.589 | Hurt |
| use latent initialization plus retrieval guidance rather than latent initialization alone (ablation) | RAG-GESTURE · BEAT2 test, all speakers | MPJPE (mm): 54.6 -> 35.0 | Helped |
| use retrieval augmentation rather than no RAG (ablation) | RAG-GESTURE · perceptual evaluation | naturalness preference: 69.6% | Helped |
| use retrieval augmentation rather than inpainting (ablation) | RAG-GESTURE · perceptual evaluation | naturalness preference: 69.6% | Helped |
Surprising findings
- Inpainting-style retrieval insertion scores lower perceptual preference than partial guidance; purely data-driven and purely retrieval-based extremes were both less preferred than the intermediate setting.
- Users showed no strong preference between LLM-based gesture-type retrieval and discourse-based retrieval (about 50/50).
Limitations
- Higher FID than seed-based single-speaker baselines (CaMN, EMAGE), attributed to those using ground-truth seeds.
- Single-speaker results are mixed; best only on diversity and L1 divergence, second-best on BeatAlign.
- Relies on an LLM and BEAT2 semantic annotations for retrieval.
Experimental setup
Datasets: BEAT2
Benchmarks: BEAT2
Metrics: FID BeatAlign L1 Divergence Diversity MPJPE user preference (naturalness, appropriateness)
Baselines: CaMN EMAGE Audio2Photoreal ReMoDiffuse Inpainting-based RAG SemanticGesticulator-style training-based RAG
Builds on: latent diffusion DDIM ReMoDiffuse SemanticGesticulator BEAT2
Closest papers by meaning
- ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis (CVPR 2024 · similarity 0.90)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.87)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.87)
- HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation (CVPR 2025 · similarity 0.87)
- Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control (AAAI 2024 · similarity 0.87)
- DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.