Research map / Video / Video generation
VBench: Comprehensive Benchmark Suite for Video Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VBench, a hierarchical benchmark suite with 16 disentangled dimensions, per-dimension prompt suites, automatic evaluation methods, and human preference annotations for evaluating text-to-video generation models.
Problem
Existing video generation metrics (FVD, IS, FID, CLIPSIM) do not align well with human perception and collapse model quality into a single number, giving little insight into specific strengths and weaknesses.
Approach
Decompose "video generation quality" hierarchically into 16 disentangled dimensions (Video Quality and Video-Condition Consistency), each with ~100 tailored prompts and a dedicated automatic evaluation method validated against human pairwise preferences per dimension.
Key results
- VBench per-dimension win ratios correlate strongly with human preference, with Spearman rho above 0.8 across dimensions (e.g., 0.9980 for Motion Smoothness, 0.9965 for Appearance Style, 0.6073 for Color being lowest).
- Evaluated LaVie, ModelScope, VideoCrafter and CogVideo across 16 dimensions; LaVie leads on Background Consistency and Temporal Flickering but has low Dynamic Degree, revealing a consistency vs dynamics trade-off.
- T2V models significantly underperform T2I models (notably SDXL) on Multiple Objects and Spatial Relationship.
Surprising findings
- Strong temporal consistency scores can be achieved by generating relatively static videos, so temporal consistency and dynamic degree trade off.
- Human category, despite being the largest share (26%) of WebVid-10M, is among the poorest-performing categories, suggesting more data does not fix complex categories.
Limitations
- Evaluates only text-to-video models; image-to-video and other tasks not yet covered.
- Does not assess safety and equality dimensions.
- Only four video generation models evaluated in the initial release.
Experimental setup
Datasets: WebVid-10M Kinetics-400 UCF-101 MSR-VTT human preference annotations
Benchmarks: VBench
Metrics: per-dimension VBench score win ratio Spearman correlation with human preference
Baselines: FVD Inception Score FID CLIPSIM Stable Diffusion 1.4 Stable Diffusion 2.1 SDXL
Builds on: CLIP DINO RAFT VideoChat
Closest papers by meaning
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.94)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.91)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.89)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.89)
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.