Research map / Video / Video generation
Evaluation of Text-to-Video Generation Models: A Dynamics Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEVIL, a dynamics-centric evaluation protocol for text-to-video models with a graded prompt benchmark, multi-granularity dynamics scores, and three dynamics metrics (range, controllability, dynamics-based quality).
Problem
Existing T2V evaluation protocols focus on temporal consistency and continuity and ignore video dynamics, which lets models score well by generating low-motion videos and fails to measure prompt-faithful dynamics.
Approach
Grade text prompts by dynamics level (GPT-4 coarse labeling plus human refinement), compute inter-frame, inter-segment, and video-level dynamics scores fused by linear regression fitted to human ratings, and derive dynamics range, controllability (ranking consistency), and dynamics-conditioned quality metrics, plus a Gemini-1.5 Pro naturalness score.
Key results
- Overall dynamics score S_v and S_f/S_s reach Pearson correlations of about 96-98% with human dynamics ratings, with the video-level score at 98.0%.
- Naturalness metric from Gemini-1.5 Pro reaches Pearson 79.0 with human ratings.
- Existing quality metrics correlate negatively with dynamics (e.g., subject consistency PC -88.9, motion smoothness PC -64.0), and GEN-2 achieves dynamics controllability 82.5 but dynamics range only 30.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DEVIL prompts instead of WebVid2M prompts and videos | Dynamics controllability evaluation · WebVid2M prompts and videos | D_control | Helped |
| Use DEVIL prompts instead of MSR-VTT prompts and videos | Dynamics controllability evaluation · MSR-VTT prompts and videos | D_control | Helped |
| Generate videos with low dynamics rather than across a broad dynamics range | GEN-2 · DEVIL benchmark | D_range: 30.8 | Hurt |
| Generate videos with low dynamics rather than across a broad dynamics range | Pika · DEVIL benchmark | D_range: 43.2 | Hurt |
| Generate a broad range of dynamics without matching prompt dynamics | FreeNoise-Lavie · DEVIL benchmark | D_control: 58.7 | Hurt |
| Generate a broad range of dynamics without matching prompt dynamics | StreamingT2V · DEVIL benchmark | D_control: 62.8 | Hurt |
| Use dynamics-level intervals to compute quality instead of averaging quality across all videos | T2V quality evaluation · DEVIL benchmark | Dynamics-based quality | Helped |
| Increase dynamics score | T2V models · Generated videos | Motion Smoothness: -64.0 | Hurt |
| Increase dynamics score | T2V models · Generated videos | Subject Consistency: -88.9 | Hurt |
| Increase dynamics score | T2V models · Generated videos | Background Consistency: -79.4 | Hurt |
| Increase dynamics score | T2V models · Generated videos | Naturalness: -51.8 | Hurt |
| Use the fitted inter-frame dynamics score S_f rather than individual inter-frame scores | DEVIL human alignment · DEVIL benchmark | Pearson correlation: 96.4 -> 96.5 | Helped |
Surprising findings
- Existing quality metrics such as subject consistency and background consistency are strongly negatively correlated with video dynamics, so top-scoring models may be rewarded for generating slow-motion videos.
- Datasets WebVid-2M and MSR-VTT achieve dynamics controllability scores of only 36.31% and 52.98% using ground-truth prompts and videos.
Limitations
- Only five dynamics grades; authors note grades should be more fine-grained.
- Only a limited number of T2V models are evaluated.
- Naturalness metric has lower human correlation (PC 79.0) and relies on a proprietary MLLM judge.
Experimental setup
Datasets: VidProm WebVid MSR-VTT DiDeMo WebVid-2M
Benchmarks: DEVIL
Metrics: Pearson correlation Kendall correlation win ratio dynamics range dynamics controllability dynamics-based quality naturalness
Baselines: VBench EvalCrafter FETV
Builds on: VBench EvalCrafter FETV RAFT ViCLIP
Closest papers by meaning
- Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality (NeurIPS 2024 · similarity 0.91)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.90)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.89)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.