Research map / Large language models / Reasoning and chain of thought
MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MME-CoT, a six-domain benchmark (1,130 questions, 3,865 key steps) and a three-aspect evaluation suite (quality via step recall/precision, robustness via stability/efficacy, efficiency via relevance rate and reflection quality) for chain-of-thought reasoning in large multimodal models.
Problem
Existing multimodal benchmarks score only final answers, so they cannot reveal whether LMM chain-of-thought is logically valid, whether CoT harms perception tasks, or how efficient long reflective reasoning is.
Approach
Evaluates the full CoT process with annotated key steps and reference captions, measuring step-level recall and precision, CoT-vs-direct accuracy differences on perception and reasoning tasks, and relevance and validity of generated steps, using GPT-4o as a step-level judge.
Key results
- QVQ-72B reaches the highest open-source CoT F1 score of 62.0%, 5.8 points above its base Qwen2-VL-72B (56.2%).
- Kimi k1.5 achieves the highest CoT quality (F1 64.2%, precision 92.0%), slightly above GPT-4o (64.0%).
- Most models show negative stability scores; InternVL2.5-8B drops 6.8 points when CoT is applied to perception tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2-VL-72B for reasoning to create QVQ | Qwen2-VL-72B · MME-CoT | F1 Score: +5.8% | Helped |
| Apply CoT prompting instead of direct prompting | InternVL2.5-8B · MME-CoT perception tasks | accuracy difference (stability): -6.8% | Hurt |
| Apply CoT prompting instead of direct prompting | Qwen2-VL-7B · MME-CoT reasoning tasks | accuracy difference (efficacy): -4.8% | Hurt |
| Apply CoT prompting instead of direct prompting | Qwen2-VL-72B · MME-CoT reasoning tasks | accuracy difference (efficacy): 2.4% | Helped |
| Apply CoT prompting instead of direct prompting | GPT-4o · MME-CoT reasoning tasks | accuracy difference (efficacy): 5.1 | Helped |
| Use the 72B model instead of the 7B model | Qwen2-VL · MME-CoT reasoning tasks | accuracy difference (efficacy): -4.8 -> 2.4 | Mixed |
| Use long CoT with reflection | QVQ-72B and Virgo-72B · MME-CoT | Reflection Quality: 61.7; 60.6 | Mixed |
| Use reflection | Kimi k1.5 · MME-CoT | Reflection Quality: 72.2 | Mixed |
Surprising findings
- CoT degrades performance on perception tasks for most models, indicating harmful overthinking.
- Long-CoT reflection models such as QVQ and Virgo reach correct answers while skipping key intermediate steps, and roughly 30-40% of their reflection steps are invalid.
Limitations
- Kimi k1.5 was evaluated on only a 150-question sample due to access limits.
- Relies on GPT-4o for annotation, step partitioning, and judging, so evaluation inherits its biases.
- Some models refuse or ignore the direct prompt, making their robustness scores unreliable.
Experimental setup
Datasets: MathVerse MMMU-Pro OlympiadBench MMT-Bench MuirBench ml-rpm-bench MMSearch CharXiv SciVerse
Benchmarks: MME-CoT
Metrics: F1 score precision recall stability efficacy relevance rate reflection quality
Baselines: GPT-4o Kimi k1.5 QVQ-72B Virgo-72B LLaVA-CoT Mulberry InternVL2.5-MPO Qwen2-VL-72B
Builds on: chain-of-thought prompting VLMEvalKit MathVerse MathVista M3CoT
Closest papers by meaning
- KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning (AAAI 2024 · similarity 0.93)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.91)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.