Research map / Video / Video generation
Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEMO, which decomposes text encoding and conditioning into content and motion parts for text-to-video diffusion, trained with text-motion and video-motion supervision on top of ModelScopeT2V.
Problem
Text-to-video diffusion models produce static or minimally dynamic videos because CLIP-style text encoders under-represent motion (verbs/adverbs) and text conditioning is spatial-only, frame by frame.
Approach
Split the text encoder into a frozen-content CLIP encoder and a fine-tuned motion encoder, add a separate temporal-transformer motion conditioning module, and supervise with cross-attention-vs-optical-flow alignment (text-motion loss), CLIP regularization, and consecutive-frame-difference matching of predicted latents (video-motion loss), using only text at inference.
Key results
- On MSR-VTT zero-shot, DEMO reaches FVD 422 vs 557 for ModelScopeT2V and FID 11.77 vs 14.89.
- On EvalCrafter, DEMO raises Flow Score from 2.51 to 4.89 and Motion AC-Score from 44 to 58 over ModelScopeT2V.
- On VBench, Motion Dynamics improves from 62.50 to 68.90, while temporal flickering and motion smoothness slightly drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · MSR-VTT | FVD: 557 -> 422 | Helped |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · MSR-VTT | CLIPSIM: 0.2941 -> 0.2965 | Helped |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · UCF-101 | FVD: 628.17 to 547.31 | Helped |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · UCF-101 | IS: 37.55 -> 36.35 | Hurt |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · EvalCrafter | Motion AC-Score: 44 -> 58 | Helped |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · EvalCrafter | VQA_T: 16.88 -> 15.65 | Hurt |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · VBench | Motion Dynamics: 62.50 to 68.90 | Helped |
| Apply DEMO's decomposed encoding, conditioning, and motion supervision | ModelScopeT2V · VBench | Motion Smoothness: 96.19 -> 96.09 | Hurt |
| Add video-motion supervision (ablation) | DEMO · EvalCrafter | Motion AC-Score: 48 -> 58 | Helped |
| Add video-motion supervision (ablation) | DEMO · EvalCrafter | Flow Score: 3.11 -> 4.89 | Helped |
| Add a motion encoder without specialized supervision or extra constraints (ablation) | ModelScopeT2V · MSR-VTT | FVD: 557 -> 552 | No clear effect |
| Train the motion encoder with text-motion supervision only, omitting regularization (ablation) | DEMO motion encoder · Motion-token sensitivity | token discrimination | Hurt |
Surprising findings
- On UCF-101, IS slightly drops (37.55 to 36.35) despite FVD improvement, attributed to over-focus on motion from short class-name prompts.
- Adding the motion encoder with only the original training loss gives performance comparable to plain fine-tuning, so the gains depend on the proposed supervision.
Limitations
- Cannot generate sequential motions specified in text; multiple actions appear simultaneously.
- Lower visual quality than LaVie and VideoCrafter2 in user study, attributed to WebVid-10M training data.
- Slight decreases in temporal flickering and motion smoothness on VBench.
Experimental setup
Datasets: WebVid-10M Kinetics-400 MSR-VTT UCF-101 EvalCrafter VBench
Benchmarks: MSR-VTT UCF-101 WebVid-10M (val) EvalCrafter VBench
Metrics: FID FVD CLIPSIM IS VQA_A VQA_T Action Score Motion AC-Score Flow Score Motion Dynamics
Baselines: ModelScopeT2V LaVie VideoCrafter2 Make-A-Video Show-1 SVD PYoCo Video LDM
Builds on: ModelScopeT2V CLIP Latent Video Diffusion Models RAFT
Closest papers by meaning
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.91)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.90)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.90)
- MotiF: Making Text Count in Image Animation with Motion Focal Loss (CVPR 2025 · similarity 0.90)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.89)
- ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.