Research map / Video / Video generation
AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AIGVQA-DB (36,576 text-to-video clips from 15 models with MOS and pairwise human ratings over four perspectives) and proposes AIGV-Assessor, an LMM-based VQA model that outputs quality levels, regression scores, and pairwise preferences.
Problem
Existing VQA and text-video metrics fail to accurately assess perceptual quality of AI-generated videos, which exhibit unique distortions such as unrealistic objects, unnatural motion, and inconsistent visual elements.
Approach
Combine spatial (InternViT) and temporal (SlowFast) encoders with projectors into an LLM, use the LLM's hidden states for quality regression across four dimensions, and fine-tune with a pairwise comparison judge trained on human preference pairs.
Key results
- On AIGVQA-DB, AIGV-Assessor reaches SRCC 0.9162 for static quality and 0.9232 for temporal smoothness, outperforming DOVER and FAST-VQA.
- On FETV, AIGV-Assessor achieves SRCC 0.936 on temporal dimension versus 0.897 for UGVQ.
- On GAIA, AIGV-Assessor improves SRCC over the best baseline DOVER by 6.7% (subject), 14.4% (interaction) and 11.65% (completeness).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIGV-Assessor instead of LIQE | AIGV quality assessment · AIGVQA-DB — Static Quality | SRCC: 0.8776 -> 0.9162 | Helped |
| Use AIGV-Assessor instead of DOVER | AIGV quality assessment · AIGVQA-DB — Temporal Smoothness | SRCC: 0.9063 -> 0.9232 | Helped |
| Use AIGV-Assessor instead of Q-Align | AIGV quality assessment · AIGVQA-DB — Dynamic Degree | SRCC: 0.5655 -> 0.6093 | Helped |
| Use AIGV-Assessor instead of DOVER | AIGV quality assessment · AIGVQA-DB — TV Correspondence | SRCC: 0.6783 -> 0.7500 | Helped |
| Use AIGV-Assessor instead of UGVQ | AIGV quality assessment · LGVQ — Spatial | SRCC: 0.759 -> 0.803 | Helped |
| Use AIGV-Assessor instead of UGVQ | AIGV quality assessment · FETV — Temporal | SRCC: 0.897 -> 0.936 | Helped |
| Use AIGV-Assessor instead of T2VQA | AIGV quality assessment · T2VQA-DB | SRCC: 0.7965 -> 0.8131 | Helped |
| Use AIGV-Assessor instead of DOVER | AIGV quality assessment · GAIA — Completeness | SRCC: 0.5198 -> 0.6635 | Helped |
| Add quality regression and LoRA fine-tuning components (ablation) | AIGV-Assessor · AIGVQA-DB — Static Quality | SRCC: 0.864 -> 0.916 | Helped |
| Add temporal features (ablation) | AIGV-Assessor · AIGVQA-DB — Temporal Smoothness | SRCC: 0.870 -> 0.923 | Helped |
Surprising findings
- Most VQA models perform better on quality evaluation than on text-video correspondence because they lack text prompt input.
- Closed-source text-to-video models exhibit much better performance than open-source models in the subjective study.
Limitations
- Text-video correspondence performance remains relatively low for all methods, including AIGV-Assessor (e.g., SRCC 0.75 on AIGVQA-DB).
- Dynamic degree prediction is the weakest dimension (SRCC 0.609).
Experimental setup
Datasets: AIGVQA-DB LGVQ FETV T2VQA-DB GAIA
Benchmarks: AIGVQA-DB LGVQ FETV T2VQA-DB GAIA
Metrics: SRCC PLCC KRCC pair accuracy
Baselines: NIQE BRISQUE CLIPScore BLIPScore AestheticScore ImageReward UMTScore Video-LLaVA Video-ChatGPT LLaVA-NeXT VideoLLaMA2 Qwen2-VL
Builds on: InternVL2 SlowFast LPIPS Q-Align DOVER FAST-VQA
Closest papers by meaning
- AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment (ICML 2025 · similarity 0.89)
- A-Bench: Are LMMs Masters at Evaluating AI-generated Images? (ICLR 2025 · similarity 0.88)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.88)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.88)
- PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild (CVPR 2024 · similarity 0.88)
- GAIA: Rethinking Action Quality Assessment for AI-Generated Videos (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.