Research map / Video / Video generation
Video-Bench: Human-Aligned Video Generation Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Video-Bench, a human-aligned text-to-video benchmark with 419 prompts and an MLLM (GPT-4o) evaluation framework using chain-of-query and few-shot scoring.
Problem
Existing video generation benchmarks either use metrics/embeddings that misalign with human judgments or LLM-based evaluators that struggle with cross-modal video-text comparison and converting textual critiques into consistent scores.
Approach
Use an MLLM evaluator that first converts video into text via multi-turn chain-of-query (LLM-generated dimension-specific questions answered by the MLLM), and scores video quality with few-shot scoring by batching videos generated from the same prompt as mutual references.
Key results
- Video-Bench achieves Spearman correlation of 0.733 on imaging quality and 0.702 on aesthetic quality with human ratings, outperforming VBench, EvalCrafter and CompBench reproductions.
- Chain-of-query improves video-condition alignment correlation by about 0.093 over single-round CompBench; few-shot scoring raises average video quality correlation from 0.561 to 0.620.
- Agreement between Video-Bench and humans averages Krippendorff's alpha 0.50, close to the human-human average of 0.52.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add few-shot scoring (ablation) | Video-Bench · Video quality | Average Spearman correlation: 0.561 -> 0.620 | Helped |
| Add chain-of-query (ablation) | Video-Bench · Video-condition alignment | Average Spearman correlation: 0.679 -> 0.7336 | Helped |
| Increase few-shot examples from 1 to 7 (ablation) | Video-Bench scoring · Video quality | Average score: 0.404 -> 0.545 | Helped |
| Use 7-shot rather than 5-shot scoring (ablation) | Video-Bench scoring · Motion Effects | Spearman correlation: 0.416 -> 0.557 | Helped |
| Use GPT-4o-1120 rather than GPT-4o-0806 | Video-Bench evaluator · Motion Effects | Spearman correlation: 0.469 -> 0.309 | Hurt |
| Use GPT-4o-1120 rather than GPT-4o-0806 | Video-Bench evaluator · Temporal Consistency | Spearman correlation: 0.494 -> 0.538 | Helped |
Surprising findings
- Human annotators showed low agreement on semantic video-condition alignment dimensions, and Video-Bench agreement with humans improved on video-text consistency.
- Newer GPT-4o snapshot (1120) performed worse than 0806 on motion effects (0.309 vs 0.469), suggesting regressions across versions.
Limitations
- Relies on proprietary GPT-4o API, limiting reproducibility and cost-free use.
- Human-agreement on some dimensions (object, color, action, scene) is low, so ground truth is noisy.
- Only 3 repeated runs for stability; only 67% exact-rating agreement across runs.
Experimental setup
Datasets: Kinetics-400 VBench prompt suite
Benchmarks: Video-Bench
Metrics: Spearman correlation Krippendorff's alpha TARa@3 mean difference with bootstrap CI
Baselines: VBench EvalCrafter CompBench MUSIQ LAION aesthetic predictor CLIP score RAFT ViCLIP UMT GRiT Tag2Text
Builds on: VBench CompBench TIFA VIEScore
Closest papers by meaning
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.94)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.93)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.89)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.