Research map / Vision-language and multimodal / Multimodal LLMs

MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMR-V, a manually annotated benchmark of 317 videos and 1,257 multiple-choice tasks testing long-range multi-frame multimodal reasoning beyond perception, and shows top models reach only 64.3%.

Problem

Existing video benchmarks mostly test perception and understanding over question-adjacent frames, not deep multimodal reasoning that requires locating and analyzing long-range evidence frames.

Approach

A video reasoning benchmark with fully human-annotated questions, implicit vs. explicit reasoning taxonomy (10 task categories, 33 subcategories), and model-aligned distractors generated by GPT-4o or written by humans, plus analyses of CoT and test-time scaling.

Key results

• Gemini-2.5-pro (1fps) achieves the best overall accuracy of 64.3% on MMR-V, versus 86.0% for humans.
• CoT yields only about 0.88% average gain and thinking models about 2.4% gain on MMR-V.
• Adding audio improves Gemini-2.0-Flash by 1.4% and Phi-4-multimodal-instruct by 1.0%.

Surprising findings

• Chain-of-thought and scaled test-time compute bring only limited gains, and visual analysis makes up only about 10% of CoTs.
• GPT-4o directly answering tasks was verified at only 17% accuracy on a 100-question pilot set.

Limitations

• Scaling the benchmark is costly because all tasks and answers are manually annotated.
• Some real-world categories such as mystery, puzzle and gaming are underrepresented.
• Most videos are in English, limiting multilingual applicability.

Experimental setup

Benchmarks: MMR-V

Metrics: accuracy

Baselines: GPT-4o Gemini-2.5-Pro GPT-5 o3 Gemini-2.0-Flash Qwen2.5-VL-72B Gemma-3-27b-it InternVL2.5-38B

Builds on: Kahneman dual process theory chain-of-thought prompting

Closest papers by meaning

• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.93)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.92)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
• Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.91)
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)