Research map / Video / Video generation
MotiF: Making Text Count in Image Animation with Motion Focal Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Motion Focal Loss (MotiF), which reweights the diffusion loss by optical-flow motion heatmaps to make a text-to-video backbone focus on moving regions for text-guided image animation, and introduces the TI2V-Bench human-evaluated benchmark.
Problem
Text-image-to-video (TI2V) models often ignore the text motion description and over-rely on the conditioning image (conditional image leakage), producing videos with weak text alignment and limited motion.
Approach
Instead of adding motion priors as inputs, add a training-objective term that weights the diffusion loss by an optical-flow-derived motion heatmap, so the model concentrates learning on high-motion regions; no extra inputs are needed at inference.
Key results
- MotiF achieves an average human preference of 72% over nine open-source TI2V models on TI2V-Bench.
- Ablation: MotiF vs. baseline without motion focal loss gives TI2V score 63.1 vs 36.9 (win/loss split); inverse-heatmap variant performs worse (61.9/38.1).
- On Animate Bench automatic metrics, MotiF gets image alignment 92.68 and text alignment 67.73, comparable to prior works rather than clearly better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Motion Focal loss using optical-flow motion heatmaps | VideoCrafter2 TI2V model · TI2V-Bench | average human preference: 72% | Helped |
| Add Motion Focal loss (ablation) | VideoCrafter2 TI2V model · TI2V-Bench | TI2V Score: 63.1/36.9 | Helped |
| Replace Motion Focal loss with inverse-motion loss (ablation) | VideoCrafter2 TI2V model · TI2V-Bench | TI2V Score: 61.9/38.1 | Hurt |
| Add cross-attention image conditioning alongside concatenation (ablation) | VideoCrafter2 TI2V model · TI2V-Bench | TI2V Score: 58.1/41.9 | Hurt |
| Replace image concatenation with cross-attention conditioning (ablation) | VideoCrafter2 TI2V model · TI2V-Bench | TI2V Score: 92.2/7.8 | Hurt |
| Use Motion Focal loss rather than standard diffusion loss alone | VideoCrafter2 TI2V model · TI2V-Bench | human preference versus prior methods: 68.8; 68.1; 84.7; 62.5; 72.5; 77.2; 66.6; 86.6; 60.3 | Helped |
Surprising findings
- A static video baseline (repeating the condition image) achieves the best image alignment score on Animate Bench, suggesting automatic alignment metrics are unreliable for TI2V.
Limitations
- Still struggles with coherent motion when multiple objects are present or new objects must enter the scene.
- Automatic metrics (image/text alignment) fail to reflect motion quality, so evaluation relies on human annotation.
- Trained on an internal licensed dataset that is not released.
Experimental setup
Datasets: internal licensed 1M video-text pairs Animate Bench
Benchmarks: TI2V-Bench Animate Bench
Metrics: TI2V score (human A-B preference) image alignment text alignment motion/justification selection rates diffusion loss ratio in high-motion regions
Baselines: DynamiCrafter VideoCrafter Cinemo I2VGen-XL SEINE Cond-leak TI2V-Zero AnimateAnything ConsistI2V
Builds on: VideoCrafter2 focal loss RAFT JUICE human evaluation protocol
Closest papers by meaning
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.90)
- TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation (AAAI 2025 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.87)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.86)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.86)
- LAMP: Learn A Motion Pattern for Few-Shot Video Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.