Research map / Video / Video generation
Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T2VHE, a standardized human evaluation protocol for text-to-video models with defined metrics, annotator training, and a dynamic evaluation module that cuts annotation cost by about half.
Problem
Human evaluation of text-to-video generation lacks reproducibility, reliability, and practicality: protocols vary widely in metrics, annotator sourcing, and annotation volume, and annotator quality is often unchecked.
Approach
Combines pairwise comparison with Rao and Kupper modeling, lightweight instruction and example training for laboratory-recruited annotators, and a dynamic module that uses automatic metric scores to order pairs and stops annotation once model rankings stabilize.
Key results
- Post-training LRAs reach Krippendorff's alpha of 0.411 on video quality versus 0.185 for pre-training LRAs with AMT, and the dynamic module cuts annotation cost to about 53% of the full protocol.
- Across 5 T2V models, closed-source Gen2 ranks first on all six metrics under AMT, post-training LRAs, and dynamic evaluation.
- Dynamic evaluation averages 1,068 annotated pairs per annotator versus 2,000 for full annotation, with rankings consistent with full annotation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instruction-based and example-based training for LRAs (ablation) | LRAs · Pilot dataset: Video Quality, Temporal Quality, Motion Quality, Text Alignment, Ethical Robustness, and Human Preference | Krippendorff's alpha (internal IAA): 0.224 -> 0.339; 0.178 -> 0.288; 0.164 -> 0.321; 0.145 -> 0.236; 0.055 -> 0.107; 0.195 -> 0.284 | Helped |
| Train LRAs with detailed guidelines and examples (ablation) | LRAs compared with AMT annotators · Video Quality, Temporal Quality, Motion Quality, Text Alignment, Ethical Robustness, and Human Preference | Krippendorff's alpha (agreement with AMT): 0.185 -> 0.411; 0.131 -> 0.340; 0.088 -> 0.338; 0.069 -> 0.327; -0.057 -> 0.100; 0.167 -> 0.281 | Helped |
| Use the dynamic evaluation module to select comparisons and stop when rankings stabilize | T2VHE · Evaluation of five text-to-video models | annotation cost: 53% | Helped |
| Use dynamic evaluation instead of full pairwise annotation | T2VHE · Evaluation of five text-to-video models | model ranking consistency | No clear effect |
| Use pre-training LRAs rather than post-training LRAs (ablation) | LRAs · Video Quality, Temporal Quality, Motion Quality, Text Alignment, Ethical Robustness, and Human Preference | Krippendorff's alpha (internal IAA): 0.339 -> 0.224; 0.288 -> 0.178; 0.321 -> 0.164; 0.236 -> 0.145; 0.107 -> 0.055; 0.284 -> 0.195 | Hurt |
| Use different prompt categories (ablation) | T2V model comparisons · Eight prompt categories | model ranking | Mixed |
Surprising findings
- Pre-training LRAs produce rankings markedly different from AMT and trained annotators, showing untrained annotators are unreliable.
- Ethical robustness has the lowest agreement even after training (alpha 0.107 for post-training LRAs).
Limitations
- Only five T2V models were evaluated, including two closed-source models, limiting generalization.
- Dynamic algorithm may frequently exclude comparisons involving the strongest model (Gen2), making its score estimates less stable.
- LRAs may be biased toward novel models and are slower than crowdsourced annotators, especially on subjective metrics.
Experimental setup
Datasets: Prompt Suite per Category
Benchmarks: Prompt Suite per Category (evaluation prompts)
Metrics: Krippendorff's alpha Rao and Kupper strength scores annotation cost reduction rank stability
Baselines: Gen2 Pika TF-T2V Latte Videocrafter2 AMT annotators pre-training LRAs
Builds on: Rao and Kupper model Krippendorff's alpha
Closest papers by meaning
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.91)
- GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning (ICML 2025 · similarity 0.89)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
- T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (NeurIPS 2024 · similarity 0.87)
- OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.