Research map / Vision-language and multimodal / Multimodal LLMs
MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVU-Eval, a 1,824-question benchmark spanning 4,959 videos and eight perception/reasoning tasks to evaluate multimodal LLMs on understanding multiple videos jointly, finding large gaps versus humans and top closed models.
Problem
Existing video understanding benchmarks take a single video as input, so MLLM ability to aggregate, compare, and reason across multiple videos (e.g., multi-camera driving, cross-angle sports analytics) is not measured.
Approach
A benchmark of multi-video QA pairs built from curated video groups (similar captions, shared objects, sports clips, etc.) with automated QA generation, leaked-information removal, filtering by MLLMs, and human verification, covering eight subtasks from perception (OR, SU, counting, comparison) to reasoning (KIR, ICL, RAG, TR).
Key results
- Gemini 2.5 Pro achieves the best overall accuracy of 58.4%, versus 93.6% for human experts.
- Qwen2.5-VL-72B reaches 57.1% overall and 77.8% on comparison, the highest comparison score; most other open-source models score below 50%.
- For VideoLLaMA3-7B, using a single video drops accuracy from 47.5% to 24.9%, and no-video input yields 16.0%, below random choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide all videos rather than one randomly sampled video (ablation) | VideoLLaMA3-7B · MVU-Eval | accuracy: 24.9 -> 47.5 | Helped |
| provide video input rather than one sampled frame per video (ablation) | VideoLLaMA3-7B · MVU-Eval | accuracy: 34.6 -> 47.5 | Helped |
| provide visual video input rather than textual descriptions (ablation) | VideoLLaMA3-7B · MVU-Eval | accuracy: 41.0 -> 47.5 | Helped |
| provide video input rather than no video input (ablation) | VideoLLaMA3-7B · MVU-Eval | accuracy: 16.0 -> 47.5 | Helped |
| use native multi-video input rather than multi-image frames per video (ablation) | Qwen2.5-VL-7B · MVU-Eval | accuracy: 45.2 -> 51.9 | Helped |
| use native multi-video input rather than concatenated merged-video input (ablation) | Qwen2.5-VL-7B · MVU-Eval | accuracy: 44.6 -> 51.9 | Helped |
| increase model size from 3B to 7B, 32B, and 72B | Qwen2.5-VL · MVU-Eval | accuracy: 46.2 -> 51.9 -> 55.6 -> 57.1 | Helped |
| increase model size from 8B to 38B | InternVL3 · MVU-Eval | accuracy: 41.7 -> 48.4 | Helped |
| increase the number of input frames (ablation) | VideoLLaMA3 · MVU-Eval | accuracy | Mixed |
| increase input resolution (ablation) | VideoLLaMA3 · MVU-Eval | accuracy | Mixed |
| increase the number of videos per question (ablation) | eight representative MLLMs · MVU-Eval questions with two to six videos | accuracy | Hurt |
| replace multi-image inputs with textual descriptions that preserve each video's process (ablation) | VideoLLaMA3-7B · MVU-Eval | accuracy: 34.6 -> 41.0 | Helped |
Surprising findings
- Some models (e.g., VideoLLaMA3-7B no-video at 16.0%) score below random chance because they refuse to answer when visual input is missing.
- Text descriptions of videos outperform multi-image (one frame per video) inputs for VideoLLaMA3-7B (41.0% vs 34.6%).
Limitations
- Videos are relatively short and do not reach movie-level lengths, limiting long-range temporal evaluation.
- Only visual inputs are supported; audio is not evaluated.
- Most videos are asynchronous, so cross-video temporal alignment is not tested extensively.
Experimental setup
Datasets: Kinetics-400 nuScenes ScanNet FineDiving YouCook2 Vchitect-2.0 DREAM-1K Kling.AI editing samples
Benchmarks: MVU-Eval
Metrics: accuracy
Baselines: Gemini 2.5 Pro Gemini 2.0 Flash Gemini 1.5 Pro GPT-4o Qwen2.5-VL-72B InternVL3-78B VideoLLaMA3-7B LLaVA-Video-7B
Builds on: Video-MME MVBench LongVideoBench MLVU
Closest papers by meaning
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.91)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.91)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos (ICLR 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.