Research map / Video / Video generation
DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptation
Zun Wang, Jialu Li, Han Lin, Jaehong Yoon, Mohit Bansal
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SR3AI region-based attention and LoRA injection (ablation) | DreamRunner (CogVideoX-2B) · DreamStorySet ablation | transition DINO: 87.1 -> 92.5 | Improved |
| add retrieval-augmented motion priors (ablation) | DreamRunner (CogVideoX-2B) · DreamStorySet ablation | fine-grained text ViCLIP: 22.5 -> 23.1 | Improved |
| add retrieval-augmented motion priors without SR3AI (ablation) | CogVideoX-2B · DreamStorySet ablation | transition DINO: 87.1 -> 84.6 | Hurt |
| add RAG on top of SR3AI (ablation) | DreamRunner · DreamStorySet ablation | transition DINO: 92.5 -> 93.6 | Mixed |
| use per-video prompts instead of a single prompt for RAG motion training (ablation) | CogVideoX-2B + RAG · 64-motion prompts | ViCLIP: 22.02 -> 23.04 | Improved |
| add RAG motion prior learning (ablation) | CogVideoX-2B · 64-motion prompts | ViCLIP: 20.84 -> 23.04 | Improved |
| add SR3A to CogVideoX-5B | CogVideoX-5B · T2V-CompBench | Dynamic-attr: 0.2250 -> 0.2832 | Improved |
| add SR3A to CogVideoX-5B | CogVideoX-5B · T2V-CompBench | Spatial: 0.5845 -> 0.6875 | Improved |
| add SR3A to CogVideoX-5B | CogVideoX-5B · T2V-CompBench | Action: 0.6040 -> 0.6340 | Improved |
| DreamRunner vs VLogger on character consistency | DreamRunner · DreamStorySet | character DINO: 41.3 -> 55.1 | Improved |
| DreamRunner vs VLogger on video smoothness | DreamRunner · DreamStorySet | video smoothness: 96.2 -> 98.1 | Improved |
| region-based LoRA injection vs global LoRA injection (ablation) | CogVideoX-2B · qualitative multi-character | visual quality | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- VLogger — Vlogger: Make Your Dream A Vlog (CVPR 2024) · better on 10 of 10 shared results
- CogVideoX-5B — CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (ICLR 2025) · better on 5 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models (NeurIPS 2025 · similarity 0.87)
- Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation (CVPR 2025 · similarity 0.87)
- StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (NeurIPS 2024 · similarity 0.86)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.86)
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.86)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
