Research map / Video / Video generation
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EvalCrafter, a 700-prompt text-to-video benchmark evaluated with 17 objective metrics across visual quality, text-video alignment, motion quality, and temporal consistency, with linear regression aligned to user ratings.
Problem
Existing text-to-video evaluation relies on distribution metrics like FVD and IS, which ignore text-video alignment, motion quality, and temporal consistency, making large T2V models hard to judge comprehensively.
Approach
Combine a diverse, metadata-annotated prompt benchmark (real user prompts plus GPT-4-generated prompts with consistency checks) with multi-aspect objective metrics, and fit linear regression coefficients to user opinion scores to produce a human-aligned leaderboard score.
Key results
- Gen2 ranks first in visual quality (62.51) and motion quality (56.43), while PikaLab ranks first in temporal consistency (65.41).
- The regression-weighted final score reaches Spearman 55.4 for visual quality vs 55.0 for simple averaging; for text-video alignment the weighted score is 32.3 vs 31.9 for averaging.
- CLIP-Score has only Spearman 6.3 correlation with human judgment, versus 26.7 for BLIP-BLEU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gen2 rather than the other evaluated models | Text-to-video generation · EvalCrafter benchmark, Visual Quality | human-aligned score: 62.51 | Helped |
| Use Show-1 rather than the other evaluated models | Text-to-video generation · EvalCrafter benchmark, Text-Video Alignment | human-aligned score: 62.07 | Helped |
| Use Gen2 rather than the other evaluated models | Text-to-video generation · EvalCrafter benchmark, Motion Quality | human-aligned score: 56.43 | Helped |
| Use PikaLab rather than the other evaluated models | Text-to-video generation · EvalCrafter benchmark, Temporal Consistency | human-aligned score: 65.41 | Helped |
| Fit human-alignment coefficients instead of directly averaging metrics | EvalCrafter evaluation · Unseen 20% samples | Spearman's ρ (visual quality): 55.0 -> 55.4 | Helped |
| Fit human-alignment coefficients instead of directly averaging metrics | EvalCrafter evaluation · Unseen 20% samples | Kendall's ϕ (visual quality): 41.0 -> 41.1 | Helped |
| Fit human-alignment coefficients instead of directly averaging metrics | EvalCrafter evaluation · Unseen 20% samples | Spearman's ρ (motion amplitude): -38.2 -> 45.0 | Helped |
| Fit human-alignment coefficients instead of directly averaging metrics | EvalCrafter evaluation · Unseen 20% samples | Kendall's ϕ (motion amplitude): -27.7 -> 32.4 | Helped |
| Use CLIP-Score rather than BLIP-BLEU as the text-video alignment metric | EvalCrafter evaluation · Human-score correlation | Spearman's ρ: 6.3 vs 26.7 | Hurt |
| Use CLIP-Score rather than BLIP-BLEU as the text-video alignment metric | EvalCrafter evaluation · Human-score correlation | Kendall's ϕ: 4.3 vs 19.0 | Hurt |
| Increase motion amplitude | Text-to-video generation · Videos preferred by users | user preference | Hurt |
| Use higher video resolution | Text-to-video generation · Visual appeal | visual quality | No clear effect |
Surprising findings
- CLIP-Score correlates very weakly with human judgment (Spearman 6.3), far below BLIP-BLEU (26.7).
- Users prefer visual appeal and slight motion over large motion amplitude, and Gen2 wins subjective likeness despite weak text alignment.
Limitations
- Only 700 prompts in the benchmark, which may not capture real-world complexity.
- Motion quality is hard to evaluate generally; metrics rely on pretrained models for action recognition and flow.
- User labels come from a small pool of annotators, possibly introducing bias.
Experimental setup
Datasets: real-world Discord prompts (FullJourney, PikaLab) DALL-Eval prompts DrawBench prompts
Benchmarks: EvalCrafter benchmark (700 prompts)
Metrics: VQA-Aesthetic VQA-Technical Inception Score CLIP-Score SD-Score BLIP-BLEU Detection-Score Count-Score Color-Score CelebrityID score
Baselines: ModelScope ZeroScope Floor33 Pictures Show-1 Hotshot-XL VideoCrafter1 Gen2 PikaLab
Builds on: FVD CLIP-Score DOVER HRS-Bench TIFA
Closest papers by meaning
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.93)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.91)
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.