Research map / Video / Video-language understanding
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMBench-Video, a long-form, multi-shot YouTube video QA benchmark with human-written free-form questions and GPT-4 3-grade scoring, evaluating open and proprietary video and image LVLMs.
Problem
Existing VideoQA benchmarks use short, mostly static videos and GPT-3.5 judging, which poorly reflect real web video use and align poorly with human judgment.
Approach
A benchmark of 609 YouTube videos (30s to 6min) with 1,998 human-written free-form QA pairs organized under a 26-leaf capability taxonomy, emphasizing temporally indispensable questions, scored by GPT-4 on a 0-3 scale.
Key results
- GPT-4o with 1fps input achieves the top overall MMBench-Video score of 2.15 (0-3 scale), versus 1.14 for the best open-source Video-LLM (LLaVA-NeXT-Video).
- GPT-4o with 1 frame retains only 47.8% of its 8-frame performance on MMBench-Video, versus about 75-93% on prior VideoQA benchmarks, indicating strong temporal indispensability.
- GPT-3.5-Turbo judges show larger MAE against human ratings than GPT-4-Turbo, which is the basis for choosing GPT-4 as the judge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add YouTube-generated subtitles | GPT-4o-[16f] · MMBench-Video subset with YouTube subtitles | overall mean: 1.90 -> 2.05 | Helped |
| Add YouTube-generated subtitles | GPT-4o-[16f] · MMBench-Video | overall mean: 1.86 -> 1.96 | Helped |
| Add YouTube-generated subtitles | GPT-4o-[8f] · MMBench-Video | overall mean: 1.62 -> 1.79 | Helped |
| Add YouTube-generated subtitles | GPT-4o without visual frames · MMBench-Video | overall mean: 0.25 -> 0.74 | Helped |
| Increase input from 1 frame to 8 frames | GPT-4o · MMBench-Video | overall mean: 0.70 -> 1.62 | Helped |
| Increase input from 8 frames to 16 frames | GPT-4o · MMBench-Video | overall mean: 1.62 -> 1.86 | Helped |
| Use 1 frame per second instead of 16 uniformly sampled frames | GPT-4o · MMBench-Video | overall mean: 1.86 -> 2.15 | Helped |
| Increase input from 1 frame to 8 frames | Idefics2-8B · MMBench-Video | overall mean: 0.95 -> 1.10 | Helped |
| Increase input from 1 frame to 8 frames | Qwen-VL-Chat · MMBench-Video | overall mean: 0.60 -> 0.52 | Hurt |
| Increase input from 1 frame to 8 frames | mPLUG-Owl2 · MMBench-Video | overall mean: 0.85 -> 1.15 | Helped |
| Increase input from 1 frame to 8 frames | InternVL-Chat-v1.5 · MMBench-Video | overall mean: 0.84 -> 1.26 | Helped |
| Use GPT-4-Turbo (1106) instead of GPT-3.5-Turbo (1106) as judge | MMBench-Video evaluation · Human-rated response subset | mean absolute error with human preferences: 0.98 -> 0.36 | Helped |
Surprising findings
- Idefics2-8B with a single image outperforms all open-source Video-LLMs on temporal reasoning.
- Adding YouTube auto-generated subtitles raises GPT-4o-[16f] overall score from 1.86 to 1.96 on the full benchmark, and from 1.90 to 2.05 on the subset with subtitles.
Limitations
- Evaluation covers a curated selection of models, not all recent high-performing ones, due to budget constraints.
- Videos are capped at 6 minutes; longer videos of tens of minutes or hours are not covered.
- Uses GPT-4 as the judge; open-source LLM judges are only partially explored.
Experimental setup
Datasets: MMBench MMStar MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA EgoSchema Video-MME Next-GQA
Benchmarks: MMBench-Video MMBench MMStar
Metrics: GPT-4 0-3 score overall mean normalized score score ratio 1-frame vs 8-frame mean absolute error vs human ratings
Baselines: Video-ChatGPT Video-LLaVA Chat-UniVi LLaMA-VID VideoChat2 MiniGPT4-Video MovieLLM PLLaVA-7B ShareGPT4Video-8B VideoStreaming LLaVA-NeXT-Video Idefics2-8B
Builds on: MMBench Video-ChatGPT evaluation protocol VLMEvalKit
Closest papers by meaning
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.95)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.93)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.93)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.92)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.92)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.