Research map / Vision-language and multimodal / Multimodal LLMs
UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether pretrained MLLMs can serve as zero-shot unified evaluators for AI-generated videos across 15 aspects, introducing the pairwise-preference UVE-Bench and benchmarking 18 MLLMs against specialized metrics and humans.
Problem
Existing AIGV automatic metrics are limited to a few specific aspects and require human-annotated data to train, making them hard to scale to finer-grained, comprehensive evaluation of modern video generative models.
Approach
Prompt pretrained MLLMs with aspect-specific guidelines to rate single videos via yes/no token probabilities or compare video pairs directly, enabling zero-shot evaluation of any AIGV aspect without training; evaluate this with a new pairwise-preference benchmark built from SOTA VGM outputs.
Key results
- Qwen2-VL-72B reaches over 80 A_single on all dynamic-degree and alignment subaspects, and InternVL-2.5-78B-MPO achieves the best single-rating average (78.2) among open models versus 88.0 human on pair comparison; Seed1.5-VL reaches 80.0 average single-rating accuracy, top overall.
- Best pair-comparison average is Gemini2.5-Flash at 54.6, versus 88.0 for human annotators, a gap of over 30 points.
- Unified MLLM evaluators match or outperform specialized metrics such as VBench and DOVER across aspects in single video rating.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the unified MLLM evaluator rather than specialized evaluation systems | AIGV evaluation on UVE-Bench · UVE-Bench single video rating | performance across evaluation aspects | Helped |
| Use detailed aspect-specific descriptions in the prompt (ablation) | InternVL-2.5-78B-MPO · UVE-Bench single video rating | A^single | Helped |
| Remove video order indicators from the prompt (ablation) | MLLM video pair comparison · UVE-Bench video pair comparison | accuracy | No clear effect |
| Use adaptive yes/no and good/bad scoring tokens rather than only yes/no (ablation) | Qwen2-VL-7B · UVE-Bench single video rating | A^single AVG | Helped |
| Generate rating scores directly as text rather than using token probabilities (ablation) | MLLM single video rating · UVE-Bench | A^single | Hurt |
| Adapt independently predicted single-video ratings into a pairwise choice (ablation) | Qwen2-VL-7B · UVE-Bench video pair comparison | accuracy | Helped |
| Adapt independently predicted single-video ratings into a pairwise choice (ablation) | Qwen2-VL-72B · UVE-Bench video pair comparison | accuracy | No clear effect |
| Increase the number of input video frames (ablation) | MLLM evaluator · UVE-Bench, Light & Color | accuracy | Helped |
| Increase the number of input video frames (ablation) | MLLM evaluator · UVE-Bench, Aesthetic Quality | accuracy | No clear effect |
| Lower beta from 0.8 to 0.7 (ablation) | Qwen2-VL-7B single video rating · UVE-Bench | A^single AVG: 70.9 -> 74.4 | Helped |
| Raise alpha from 0.4 to 0.5 (ablation) | Qwen2-VL-7B single video rating · UVE-Bench | A^single AVG: 70.9 -> 72.5 | Helped |
| Use positive instead of negative prompts (ablation) | Qwen2-VL-7B · UVE-Bench, Dynamic Degree | A^single: 66.0 -> 84.6 | Helped |
Surprising findings
- Removing video order indicators barely changes pairwise accuracy, unlike findings for image comparison.
- Video-LLaVA and VideoChat2-Mistral, with lower average scores, outperform 72B models on motion naturalness, which the authors attribute to native video encoders.
Limitations
- Current MLLMs, even the largest, lag far behind humans especially on temporal quality and motion alignment.
- Annotation by the paper's authors (small annotator pool) with a limited set of 1,230 videos and 4,042 pair annotations.
- Only text-to-video evaluation; image-to-video, safety, and bias aspects are not covered.
Experimental setup
Datasets: VideoGen-Eval Movie Gen Video Bench ShutterStock TempCompass
Benchmarks: UVE-Bench
Metrics: A_single accuracy (rating consistency with human preference) four-way pair comparison accuracy
Baselines: VBench VideoScore-v1.1 UMTScore VIDEOCON-PHYSICS DOVER Random
Builds on: VBench VideoScore DOVER UMTScore VIDEOCON-PHYSICS
Closest papers by meaning
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.89)
- Is Your Video Language Model a Reliable Judge? (ICLR 2025 · similarity 0.88)
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs (NeurIPS 2025 · similarity 0.88)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.88)
- AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.