Research map / Video / Video generation
GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GRADEO-Instruct, a 3.3k-video multi-dimensional T2V evaluation instruction dataset with human-annotated rationales converted to CoT by GPT-4o, and fine-tunes Qwen2-VL-7B (LoRA) into GRADEO, an explainable multi-step reasoning video evaluator that correlates better with humans than MLLMs and metrics.
Problem
Automated text-to-video evaluation metrics lack high-level semantic understanding, reasoning, and explainability, and general MLLMs are not aligned with human video assessment across diverse dimensions.
Approach
Train an MLLM evaluator on human-annotated video rationales and scores, converted into a four-step chain-of-thought (Overview, Description, Analysis, Assessment) so the model reasons before scoring across seven dimensions from low-level quality to high-level safety, rationality, and creativity.
Key results
- GRADEO reaches SROCC 0.743 on Quality and 0.797 on Creativity on its test set, the best across all seven dimensions versus image and video MLLMs and metrics.
- On pairwise preference accuracy GRADEO scores 0.815 on T2VQA-DB, 0.790 on GenAI-Bench-Video, 0.780 on TVGE alignment and 0.840 on TVGE quality, above VideoScore-v1.1 (0.790, 0.760, 0.690, 0.795).
- Ablation: full CoT (Overview + Description + Analysis + Assessment) fine-tuning gives SROCC 0.692 versus 0.538 for plain fine-tuning without reasoning steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Quality | SROCC: 0.617 -> 0.743 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Aesthetic | SROCC: 0.663 -> 0.717 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Consistency | SROCC: 0.575 -> 0.634 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Alignment | SROCC: 0.508 -> 0.601 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Rationality | SROCC: 0.580 -> 0.606 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Safety | SROCC: 0.663 -> 0.747 | Helped |
| Use GRADEO instead of the strongest baseline | Video evaluation on the GRADEO-Instruct test set · Creativity | SROCC: 0.792 -> 0.797 | No clear effect |
| Use GRADEO instead of the strongest baseline | Pairwise video evaluation · T2VQA-DB Video Fidelity | pairwise accuracy: 0.790 -> 0.815 | Helped |
| Use GRADEO instead of the strongest baseline | Pairwise video evaluation · GenAI-Bench-Video Video Preference | pairwise accuracy: 0.760 -> 0.790 | Helped |
| Use GRADEO instead of the strongest baseline | Pairwise video evaluation · TVGE Text-Video Alignment | pairwise accuracy: 0.730 -> 0.780 | Helped |
| Use GRADEO instead of the strongest baseline | Pairwise video evaluation · TVGE Video Quality | pairwise accuracy: 0.795 -> 0.840 | Helped |
| Add CoT reasoning without fine-tuning (ablation) | Qwen2-VL-7B video evaluator · GRADEO-Instruct test set | mean SROCC: 0.512 -> 0.572 | Helped |
Surprising findings
- Video MLLMs did not outperform image MLLMs on this video evaluation task.
- Models with stronger instruction following and more training data are more prone to generating unsafe content, per the benchmarking insights.
Limitations
- Training data is relatively small (about 2.9k training samples) and rationales are GPT-4o-generated from human notes.
- Benchmark uses only 8 open-source T2V models, with videos short (1-5 s).
- Some baselines like GPT-4o refuse to answer in some cases, affecting comparisons.
Experimental setup
Datasets: GRADEO-Instruct WebVid VidProM SafeSora T2VQA-DB GenAI-Bench-Video TVGE
Benchmarks: GRADEO-Instruct test split T2VQA-DB GenAI-Bench-Video TVGE
Metrics: Spearman rank correlation (SROCC) Pearson correlation (PLCC) mean absolute error pairwise preference accuracy
Baselines: PIQE BRISQUE CLIP-Score ImageReward HPS-v2.1 LLaVA-1.5-7B LLaVA-1.6-7B GLM-4v GPT-4-Turbo GPT-4o InternVL2.5-8B LLaVA-NeXT-Video-7B
Builds on: Qwen2-VL LoRA Chain-of-Thought prompting VBench VideoScore
Closest papers by meaning
- Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality (NeurIPS 2024 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.88)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.87)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
- VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation (NeurIPS 2025 · similarity 0.87)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.