Research map / Video / Video generation
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamVideo, which customizes both subject identity and motion for text-to-video generation by decoupling learning into a textual-inversion-plus-identity-adapter subject stage and a motion adapter trained with appearance guidance, on a frozen ModelScopeT2V backbone.
Problem
Customized video generation requires controlling both the subject's appearance and the temporal motion, but prior methods customize only one of these aspects.
Approach
Decouple subject learning (textual inversion followed by a lightweight identity adapter in cross-attention layers) from motion learning (a temporal motion adapter conditioned on a CLIP image embedding of a random training frame to suppress appearance leakage), with the pretrained video diffusion model frozen throughout.
Key results
- On joint subject+motion customization, DreamVideo reaches CLIP-T 0.314 and DINO-I 0.452 with 85M parameters, versus AnimateDiff 0.298/0.432 at 1.21B.
- Motion customization: CLIP-T 0.309 vs ModelScopeT2V 0.293 and Tune-A-Video 0.290.
- Subject customization: DINO-I 0.475 vs Dreamix 0.459 with 11M parameters vs Dreamix 823M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add textual identity learning (ablation) | DreamVideo · Joint subject and motion customization | CLIP-I: 0.657 -> 0.665 | Helped |
| Add textual identity learning (ablation) | DreamVideo · Joint subject and motion customization | CLIP-T: 0.310 -> 0.314 | Helped |
| Add textual identity learning (ablation) | DreamVideo · Joint subject and motion customization | DINO-I: 0.435 -> 0.452 | Helped |
| Add textual identity learning (ablation) | DreamVideo · Joint subject and motion customization | Temporal Consistency: 0.968 -> 0.971 | Helped |
| Add the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | CLIP-T: 0.295 -> 0.314 | Helped |
| Add the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | Temporal Consistency: 0.964 -> 0.971 | Helped |
| Add appearance guidance to the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | DINO-I: 0.421 -> 0.452 | Helped |
| Add appearance guidance to the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | CLIP-I: 0.650 -> 0.665 | Helped |
| Add appearance guidance to the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | Temporal Consistency: 0.970 -> 0.971 | Helped |
| Add appearance guidance to the motion adapter (ablation) | DreamVideo · Joint subject and motion customization | CLIP-T: 0.305 -> 0.314 | Helped |
Surprising findings
- Training AnimateDiff's motion module from scratch on the customization setting led to unstable results, so the authors fine-tuned its pretrained weights instead.
- AnimateDiff achieves the highest temporal consistency but generates videos with small motions.
Limitations
- Fails to generate videos with multiple subjects and multiple motions simultaneously.
- Requires separate training of identity and motion adapters per subject and motion, and no general customized video model is provided.
Experimental setup
Datasets: UCF101 UCF Sports Action DAVIS
Benchmarks: DreamVideo subject and motion customization set (20 subjects, 30 motions, 42 prompts)
Metrics: CLIP-T CLIP-I DINO-I Temporal Consistency user study preference
Baselines: AnimateDiff ModelScopeT2V LoRA Textual Inversion Dreamix Tune-A-Video
Builds on: Textual Inversion ModelScopeT2V DreamBooth AnimateDiff LoRA
Closest papers by meaning
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.89)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.89)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.87)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.86)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.86)
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.