Research map / Video / Video-language understanding
VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VimoRAG, which retrieves human-centric videos from a 425K-video database to augment text-conditioned motion LLM generation, using a Gemini-MVR retriever and McDPO (dual-alignment DPO) to handle noisy retrievals.
Problem
Motion LLMs suffer OOD/OOV generalization issues because text-motion datasets are small (~14k samples); existing motion RAG retrieves only from 3D motion databases, and retrieval errors propagate into generation.
Approach
Retrieve 2D human-centric videos from large in-the-wild video databases via a text-to-video retriever combining action-level (keypoint-based) and object-level (InternVideo) channels with a linear integrator, then feed the retrieved video and text into a Phi-3 motion LLM trained with a DPO variant whose preference pairs are built from a motion-and-text dual-alignment reward to tolerate poor retriev
Key results
- On HumanML3D, VimoRAG (Phi3-3.8B) reaches FID 0.131 vs 0.501 for MotionGPT with the same backbone, with R-Precision Top-1 0.452 vs 0.396.
- On zero-shot IDEA400, VimoRAG achieves FID 2.388, the best among all compared models, versus 5.544 for MotionGPT (Motion LLM).
- Gemini-MVR improves text-to-video R@1 from 53.6 to 58.3 on human-centric videos over InternVideo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video-based retrieval augmentation and McDPO | MotionGPT with Phi3-3.8B backbone · HumanML3D test | FID: 0.501 -> 0.131 | Helped |
| add video-based retrieval augmentation and McDPO | MotionGPT with Phi3-3.8B backbone · HumanML3D test | R-Precision Top 1: 0.396 -> 0.452 | Helped |
| add video-based retrieval augmentation and McDPO | MotionGPT with Phi3-3.8B backbone · HumanML3D test | Diversity: 9.475 -> 9.424 | Hurt |
| add video-based retrieval augmentation and McDPO | MotionGPT · IDEA400 test | FID: 5.544 -> 2.388 | Helped |
| replace Gemini-MVR with random retrieval (ablation) | VimoRAG with McDPO · HumanML3D validation | FID: 0.148 -> 0.544 | Hurt |
| replace Gemini-MVR with InternVideo (ablation) | VimoRAG with McDPO · HumanML3D validation | FID: 0.148 -> 0.205 | Hurt |
| add McDPO (ablation) | VimoRAG with Gemini-MVR · HumanML3D validation | FID: 0.260 -> 0.148 | Helped |
| replace InternVideo with Gemini-MVR | video retrieval on HcVD human-centric video test · HcVD test: human-centric video | R@1: 53.6 -> 58.3 | Helped |
| replace InternVideo with Gemini-MVR | video retrieval on HcVD single human-centric video test · HcVD test: single human-centric video | R@1: 52.3 -> 61.0 | Helped |
| increase retrieval database size | VimoRAG · HumanML3D validation | R-Precision | Helped |
| increase retrieval database size | VimoRAG · HumanML3D validation | Diversity | No clear effect |
| increase LoRA rank and alpha while keeping alpha/rank=2 (ablation) | McDPO · HumanML3D validation | FID: 0.417 -> 0.179 | Helped |
Surprising findings
- Random video retrieval during inference yields distributions comparable to rank-1 retrieval, suggesting robustness to retrieval failure.
- Larger retrieval databases increase R-Precision but do not consistently increase diversity.
Limitations
- Inference latency is dominated by the 3.8B LLM generation step, slower than small motion specialists.
- Gains over motion specialists are not uniform; R-Precision and MM Dist remain behind some specialists on HumanML3D.
- Only the rank-1 retrieved video is used and the LLM backbone choice was not explored.
Experimental setup
Datasets: HumanML3D IDEA400 (Motion-X subset) HcVD (UCF101, NTU RGB+D, ASLAN, HMDB51, Kinetics-400, PennAction, Motion-X)
Benchmarks: HumanML3D IDEA400
Metrics: FID R-Precision MM Dist Diversity R@1 R@5 R@10 MnR
Baselines: MoMask T2M-GPT MDM MotionDiffuse MLD MotionGPT ReMoDiffuse MotionGPT-2
Builds on: ReMoDiffuse MotionGPT LLaVA-style visual instruction tuning DPO InternVideo
Closest papers by meaning
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.91)
- Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · similarity 0.89)
- LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning (ICLR 2025 · similarity 0.88)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.87)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.87)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.