Research map / Video / Video-language understanding
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMVU, a 3,000-question expert-annotated multi-discipline video benchmark across 27 subjects, and evaluates 36 multimodal models with CoT and direct answering.
Problem
Existing multimodal benchmarks rarely test expert-level, knowledge-intensive reasoning over specialized-domain videos; prior video benchmarks are mostly general-purpose or rely on text/image or auto-generated items.
Approach
A textbook-guided pipeline where experts find textbook concepts, source CC-licensed silent vision-intensive YouTube videos, and write questions with distractors plus expert-annotated domain knowledge and reasoning rationales, followed by expert validation that video is required to answer.
Key results
- o1 and Gemini 2.0 Flash Thinking achieve the top results on MMVU test (75.5% and 69.1% average), while the best human open-book score is 86.8%.
- GPT-4o reaches 66.7% test accuracy with CoT, and CoT generally improves over direct answering, with Claude 3.5 Sonnet gaining 11.0%.
- Qwen2-VL-72B and DeepSeek-VL2 exceed human closed-book accuracy (49.7%) in the CoT setting, per the authors' test-set results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Chain-of-Thought reasoning instead of direct answering | multimodal foundation models · MMVU validation set | accuracy | Helped |
| use System-2 thinking and long Chain-of-Thought | multimodal foundation models · MMVU | accuracy | Helped |
Surprising findings
- Models often rely on textual information over the video, with heavy reliance on text accounting for 20% of sampled errors.
- CoT gains are not uniform: GPT-4o improves only marginally while Claude 3.5 Sonnet gains substantially.
Limitations
- Test set is hidden and evaluated via a planned online pipeline, limiting open inspection.
- CC-licensed restriction excludes some subjects such as sports and may limit coverage.
- o1 was evaluated on a sampled subset (100 validation, 200 test) because multimodal API access was not granted.
Experimental setup
Datasets: MMVU
Benchmarks: MMVU
Metrics: accuracy
Baselines: GPT-4o o1 Gemini 2.0 Flash Thinking Gemini 2.0 Flash Claude 3.5 Sonnet Qwen2-VL-72B DeepSeek-VL2 InternVL2.5-38B
Builds on: MMMU MMMU-Pro MMWorld Video-MME
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- MLVU: Benchmarking Multi-task Long Video Understanding (CVPR 2025 · similarity 0.89)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.89)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.88)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.87)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.