Research map / Video / Video-language understanding
Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Q-Bench-Video, a benchmark of 2,378 QA pairs (MCQ Yes-or-No/What-How, open-ended, and video-pair comparisons) over natural, AIGC, and CG videos to test video quality understanding of 17 LMMs, which all trail human performance.
Problem
Existing LMM video benchmarks focus on semantic understanding and neglect video quality perception (technical, aesthetic, temporal, and AIGC distortions).
Approach
A quality-focused video benchmark built from VQA datasets with uniform quality sampling, multiple question formats including open-ended and video-pair comparison questions, and a new AIGC distortion dimension.
Key results
- GPT-4o achieves the best overall accuracy at 58.70%, still 22.86 points below human performance (81.56%).
- Open-source mPLUG-Owl3 reaches 52.39% overall, slightly above GPT-4o mini (52.20%).
- LMMs perform better on video-pair comparison (e.g., Gemini 1.5 Pro 63.55% pair overall) than single-video analysis, while open-ended questions are hardest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an LMM rather than random guessing for video-quality questions | LMMs evaluated on Q-Bench-Video · Q-Bench-Video test subset | overall accuracy: 37.79% (random guess) -> 52.39% (best open-source LMM) | Helped |
| Use human judgment rather than the best-performing LMM | Video-quality understanding · Q-Bench-Video test subset | overall accuracy: 58.70% -> 81.56% | Helped |
| Use proprietary rather than open-source LMMs | LMM video-quality understanding · Q-Bench-Video test subset | overall accuracy: 52.39% -> 58.70% | Helped |
| Ask open-ended rather than Yes-or-No questions | LMM video-quality assessment · Q-Bench-Video test subset | accuracy: 67.48% -> 49.25% | Hurt |
| Ask What-How rather than Yes-or-No questions | GPT-4o · Q-Bench-Video test subset | accuracy: 67.48% -> 58.79% | Hurt |
| Evaluate AIGC distortions rather than aesthetic distortions | Human video-quality assessment · Q-Bench-Video test subset | accuracy: 80.23% -> 86.21% | Helped |
| Evaluate aesthetic distortions rather than AIGC distortions | LMM video-quality assessment · Q-Bench-Video test subset | accuracy: 52.22% -> 58.57% | Helped |
| Compare video pairs rather than assess single videos | GPT-4o · Q-Bench-Video | overall accuracy: 56.31% -> 63.80% | Helped |
| Compare coarse-grain rather than fine-grain video pairs | GPT-4o · Q-Bench-Video video pairs | accuracy: 67.30% -> 69.24% | Helped |
Surprising findings
- Open-source LMMs such as mPLUG-Owl3 nearly match GPT-4o mini overall.
- LMMs are notably weaker on AIGC distortions than on aesthetic distortions, while humans excel at AIGC distortions.
Limitations
- Annotation subjectivity, especially for aesthetic quality.
- Benchmark may become outdated as new video generation models produce different distortions.
- Correct answers for dev and test subsets are withheld and proprietary.
Experimental setup
Datasets: LSVQ MaxWell WaterlooSQoE-III WaterlooSQoE-IV T2VQA-DB VideoFeedback LIVE-YT-Gaming
Benchmarks: Q-Bench-Video
Metrics: accuracy GPT-assisted score for open-ended
Baselines: Random guess Human GPT-4o Gemini 1.5 Pro Gemini 1.5 Flash GPT-4 Turbo GPT-4o mini LLaVA-OneVision mPLUG-Owl3 InternVL-Chat
Builds on: LSVQ MaxWell T2VQA-DB LIVE-YT-Gaming
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.89)
- VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation (CVPR 2025 · similarity 0.88)
- VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment (AAAI 2025 · similarity 0.88)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.88)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.