Research map / Video / Video-language understanding
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVBench, a 20-task multiple-choice video benchmark built by converting static image tasks to temporal ones and auto-generating QAs from public video annotations, plus a VideoChat2 baseline trained with progressive multimodal training.
Problem
Existing multimodal LLM benchmarks mostly test spatial understanding on static images, leaving temporal understanding in videos under-evaluated; prior video benchmarks are narrow or rely on labor-intensive annotation or biased LLM scoring.
Approach
A static-to-dynamic method that turns static image-benchmark task definitions into 20 temporal video tasks, with multiple-choice QAs automatically generated from ground-truth annotations of 11 public video datasets so evaluation avoids LLM judging.
Key results
- VideoChat2 reaches 51.1% average accuracy on MVBench, over 15 points above the best prior video MLLM VideoChat (35.5%).
- Existing image and video MLLMs score roughly 18.8% to 36.0% average, close to random chance at 27.3%.
- Replacing the visual encoder with UMT-L adds about 6.2 points over EVA-CLIP-g (48.6 vs 42.4 without LoRA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train VideoChat2 instead of using LLaVA | Vicuna-7B video MLLM · MVBench | accuracy: 36.0 -> 51.1 | Helped |
| Use VideoChat2 instead of VideoChatGPT | Video MLLM · Video conversation benchmark | average score: 2.38 -> 2.98 | Helped |
| Use VideoChat2 instead of VideoChatGPT | Video MLLM · MSVD-QA | accuracy: 64.9 -> 70.0 | Helped |
| Use VideoChat2 instead of VideoChatGPT | Video MLLM · MSRVTT-QA | accuracy: 49.3 -> 54.1 | Helped |
| Use VideoChat2 instead of VideoChatGPT | Video MLLM · ANet-QA | accuracy: 35.2 -> 49.1 | Helped |
| Use video instruction data instead of image-only instruction data (ablation) | VideoChat2 · MVBench | accuracy: 42.1 -> 50.5 | Helped |
| Compress simple caption training data (ablation) | VideoChat2 · MVBench | accuracy: 51.1 -> 50.7 | Hurt |
| Replace EVA-CLIP-g with UMT-L as the visual encoder (ablation) | VideoChat2 with Vicuna-7B v0 · MVBench | accuracy: 42.4 -> 48.6 | Helped |
| Add LoRA tuning (ablation) | VideoChat2 with UMT-L and Vicuna-7B v0 · MVBench | accuracy: 48.6 -> 51.1 | Helped |
| Replace Vicuna-7B v0 with Vicuna-13B v0 (ablation) | VideoChat2 with UMT-L · MVBench | accuracy: 48.6 -> 51.4 | Helped |
| Replace Vicuna-7B v1.5 with Vicuna-13B v1.5 (ablation) | VideoChat2 with UMT-L · MVBench | accuracy: 51.2 -> 51.6 | Helped |
| Tune QFormer in Stage 2 (ablation) | VideoChat2 · MVBench | accuracy: 38.5 -> 47.0 | Helped |
Surprising findings
- VideoChat2 run with blank video inputs (VideoChat2text) scores 34.7%, nearly matching VideoChat (35.5%), indicating the prior video MLLM barely uses temporal visual content.
- Scaling the LLM from Vicuna-7B to 13B gives only marginal gains (51.1 to 51.4), suggesting MVBench relies mainly on the visual encoder.
Limitations
- Evaluation uses only multiple-choice QA, which may not capture open-ended generation quality.
- VideoChat2 struggles on position, count, and character tasks, performing below its blank-video variant on those.
- Benchmark questions are generated automatically via ChatGPT, which may introduce noise in question and option quality.
Experimental setup
Datasets: STAR PAXION Moments in Time V1 FunQA CLEVRER Perception Test Charades-STA MoVQA NTU RGB+D VLN-CE TVQA VQAv2
Benchmarks: MVBench MSVD-QA MSRVTT-QA ANet-QA VideoChatGPT video conversation benchmark
Metrics: accuracy average accuracy across 20 tasks ChatGPT-judged score for conversation
Baselines: VideoChat VideoChatGPT VideoLLaMA LLaVA InstructBLIP mPLUG-Owl Otter MiniGPT-4 BLIP2 LLaMA-Adapter
Builds on: BLIP-2 UMT VideoChat VideoChatGPT MMBench
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.95)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.94)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.92)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.92)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.91)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.