Research map / Video / Video generation
MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionRAG, which retrieves text-matched reference videos, adapts their motion features to a target image with an in-context causal transformer (CAMA), and injects them into frozen image-to-video diffusion models via an attention adapter.
Problem
Image-to-video diffusion models often generate weak or unrealistic motion, since text prompts lack the precise temporal and kinematic detail that real video examples provide, and compositional motion generalization is limited.
Approach
Retrieve semantically matched videos by caption, extract high-level VideoMAE motion tokens, and use a causal in-context transformer over reverse-ordered (first frame, motion) examples to predict motion tokens for the target image, then inject them through a motion adapter into frozen I2V backbones, with no per-instance fine-tuning.
Key results
- On OpenVid-1K, DynamiCrafter+RAG raises Action score from 53.5 to 62.1 and reduces FVD from 88.4 to 69.0.
- CogVideoX+RAG reaches Action 65.8 (vs 59.9 baseline) and FVD 80.2 (vs 87.1).
- Zero-shot transfer to SkillVid improves Action for all three models, e.g. CogVideoX 51.5 to 53.5, with added inference time under 4 seconds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MotionRAG | CogVideoX-5b · OpenVid-1K | Action: 59.9 -> 65.8 | Helped |
| Add MotionRAG | CogVideoX-5b · OpenVid-1K | FVD: 87.1 -> 80.2 | Helped |
| Add MotionRAG | Dynamicrafter · OpenVid-1K | Action: 53.5 -> 62.1 | Helped |
| Add MotionRAG | Dynamicrafter · OpenVid-1K | FVD: 88.4 -> 69.0 | Helped |
| Add MotionRAG | Stable Video Diffusion · OpenVid-1K | FVD: 98.0 -> 167.1 | Hurt |
| Use the Motion Context Transformer with 9 references instead of averaging 9 retrieved features (ablation) | Dynamicrafter · OpenVid-1K | Action: 57.9 -> 62.1 | Helped |
| Use the Motion Context Transformer with 9 references instead of averaging 9 retrieved features (ablation) | Dynamicrafter · OpenVid-1K | FVD: 78.7 -> 69.0 | Helped |
| Increase Motion Context Transformer references from 5 to 9 (ablation) | Dynamicrafter · OpenVid-1K | Action: 61.0 -> 62.1 | Helped |
| Use a single random retrieved video as context (ablation) | Dynamicrafter · OpenVid-1K | Action: 53.5 -> 52.2 | Hurt |
| Use Motion Context Transformer with 9 random videos as context (ablation) | Dynamicrafter · OpenVid-1K | Action: 53.5 -> 60.7 | Helped |
| Switch retrieval database from OpenVid-1M to InternVid-10M (ablation) | MotionRAG with Dynamicrafter · OpenVid-1K | Action: 62.1 -> 60.9 | Hurt |
Surprising findings
- SVD+RAG gets a higher FVD than the SVD baseline on OpenVid-1K (167.1 vs 98.0) despite better Action, CLIP, and DINO scores.
- Random retrieval with the MCT (MCT-Rand-9) still improves Action from 53.5 to 60.7, suggesting the adapter relies on learned motion priors rather than the retrieved content.
Limitations
- Conflicting motions in retrieved references can cancel out, producing nearly static outputs (e.g., averaging upward and downward jump phases).
- Restricted to short clips because it depends on base models' temporal capacity.
- Fine-grained motions such as subtle hand gestures are not well captured given the current retrieval corpus and video encoder.
Experimental setup
Datasets: OpenVid-1M SkillVid InternVid-10M Something-Something v2
Benchmarks: OpenVid-1K SkillVid
Metrics: Action score CLIP score DINO score FVD FID inference time
Baselines: Stable Video Diffusion DynamiCrafter CogVideoX-5b Top-1 retrieval Avg-9 feature averaging Motion-I2V search-T2V
Builds on: IP-Adapter VideoMAE DINOv2 Sentence-BERT search-T2V
Closest papers by meaning
- DisMo: Disentangled Motion Representations for Open-World Motion Transfer (NeurIPS 2025 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.88)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.