Research map / Video / Video-language understanding
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Video-MME, a manually annotated 900-video, 2,700-question multiple-choice benchmark spanning 6 domains, 30 subfields and 11s to 1h durations, with subtitle and audio inputs, to evaluate MLLMs on video understanding.
Problem
Existing video benchmarks for multimodal LLMs lack diversity in video types, temporal coverage across durations, and multimodal inputs (subtitles, audio), so they do not comprehensively assess MLLM video understanding.
Approach
A full-spectrum video QA benchmark with expert-annotated 3 questions per video across short/medium/long durations, subtitles and audio tracks, and a text-only filter using Gemini 1.5 Pro to remove questions answerable without video.
Key results
- Gemini 1.5 Pro achieves 75.0% overall accuracy without subtitles (81.3% with subtitles), the best commercial model, versus 71.9% for GPT-4o.
- Best open-source model VILA-1.5 (34B) reaches 59.0% overall accuracy, with VITA-1.5 at 56.1% among 7B models.
- Adding subtitles improves Gemini 1.5 Pro by 6.2 points overall and audio by 4.3 points; accuracy declines from short to long videos for all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add subtitles to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME overall | accuracy: +6.2% | Helped |
| add audio to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME overall | accuracy: +4.3% | Helped |
| add subtitles to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME short videos | accuracy: +2.8% | Helped |
| add subtitles to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME long videos | accuracy: +10.1% | Helped |
| add subtitles to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME | accuracy: +6.2% | Helped |
| add audio to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME long videos | accuracy: +6.2% | Helped |
| add subtitles to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME Multilingual long videos | accuracy: +16.7% | Helped |
| add audio to Gemini 1.5 Pro's frame input (ablation) | Gemini 1.5 Pro · Video-MME Multilingual long videos | accuracy: +12.5% | Helped |
| increase video duration from short to long | Gemini 1.5 Pro · Video-MME | accuracy: −14.3% | Hurt |
| add subtitles (ablation) | Qwen-VL-Max · Video-MME overall | accuracy: 51.3 -> 51.2 | Hurt |
| add subtitles (ablation) | VILA-1.5 · Video-MME medium videos | accuracy: 58.1 -> 57.4 | Hurt |
| add subtitles (ablation) | Gemini 1.5 Pro · Video-MME Life Record short videos | accuracy: 88.1 -> 86.6 (-1.5) | Hurt |
Surprising findings
- Image MLLMs such as Qwen-VL-Max and InternVL-Chat-V1.5 with multi-frame inputs reach around 50% accuracy, close to the video-specific LLaVA-NeXT-Video.
- Gemini 1.5 Pro achieves less than 15% accuracy on text-only questions, indicating questions require video content.
Limitations
- Most open-source models use a fixed small number of frames (e.g., 8), causing sparse information for long videos.
- Most open-source models do not accept audio input, so audio evaluation was omitted for them.
Experimental setup
Datasets: Video-MME
Benchmarks: Video-MME
Metrics: accuracy
Baselines: GPT-4V GPT-4o Gemini 1.5 Flash VILA-1.5 LLaVA-NeXT-Video VideoChat2-Mistral InternVL-Chat-V1.5 Qwen-VL-Max
Builds on: EgoSchema MVBench TempCompass
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.96)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.93)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.92)
- ALLVB: All-in-One Long Video Understanding Benchmark (AAAI 2025 · similarity 0.92)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.92)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.