Research map / Video / Video-language understanding
MLVU: Benchmarking Multi-task Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLVU, a multi-task long video understanding benchmark with 3,102 QA pairs across 9 tasks and diverse genres and lengths (3 min to 2 hours), and evaluates 23 MLLMs in zero-shot.
Problem
Existing video understanding benchmarks use short videos, cover few genres and tasks, and often contain questions answerable without long-video content, so they inadequately evaluate long video understanding of MLLMs.
Approach
A benchmark with long videos (3 min to 2 hours, avg ~15 min), diverse genres, and nine LVU-oriented tasks (holistic, single-detail, multi-detail), where local-information questions have unambiguous referring context and needle-style insertion is used to construct tasks.
Key results
- GPT-4o achieves an M-Avg of 54.5% on MLVU test multiple-choice tasks and a G-Avg of 5.87 on generation tasks, the best among 23 evaluated MLLMs.
- LLaVA-OneVision trails GPT-4o by only 2.8% in M-Avg among open-source long video MLLMs.
- Performance of all models declines as video length grows (180s, 360s, 600s in MLVU Time-ladder); most models fail on action order and action count tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase video input from 16 to 90 frames (ablation) | MiniGPT4-Video · MLVU | M-Avg: 24.2 -> 31.7 | Helped |
| increase video input from 16 to 256 frames (ablation) | GPT-4o · MLVU | M-Avg: 45.8 -> 54.5 | Helped |
| use stronger image-understanding capability (ablation) | MLLMs · MLVU | M-Avg: 17.1 -> 43.3 | Helped |
| replace Vicuna-7B with Vicuna-13B backbone (ablation) | Video-LLaMA-2 · MLVU | M-Avg: 13.3 -> 18.8 | Helped |
| replace LLaMA-7B with Mistral-7B backbone (ablation) | MiniGPT4-Video · MLVU | M-Avg: 20.6 -> 31.7 | Helped |
| increase video duration (ablation) | MLLMs · MLVU Time-ladder | average accuracy | Hurt |
| increase the number of action probes (ablation) | MLLMs · MLVU Action Count | accuracy | Hurt |
| move the referring clip to different positions in the video (ablation) | recent long-video MLLMs · MLVU Ego Reasoning and Plot QA | accuracy | No clear effect |
Surprising findings
- Most models, except GPT-4o and Video-XL, fail entirely on action order and action count tasks.
- GPT-4o reaches only 42.9% on needle question-answering, far below text-domain needle-in-a-haystack performance of long LLMs.
Limitations
- Many evaluated models are proprietary or evaluated via APIs, limiting reproducibility of some results.
- Some tasks rely on GPT-4 judging for generation quality (VS, SSC).
Experimental setup
Datasets: ULVC UCF Crime WebVid Clevrer Ego4D Kinetics ActivityNet-Captions Movie101 MovieNet
Benchmarks: MLVU
Metrics: accuracy GPT-4 judged score (0-10)
Baselines: GPT-4o LLaVA-OneVision VideoLLaMA2 LongVA Video-XL InternVL-2 MiniGPT4-Video Video-CCAM MA-LMM MovieChat
Builds on: NIAH (Needle-In-A-Haystack) MovieChat MVBench EgoSchema
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
- ALLVB: All-in-One Long Video Understanding Benchmark (AAAI 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.89)
- MMVU: Measuring Expert-Level Multi-Discipline Video Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.