Research map / Large language models / Reasoning and chain of thought

M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces M3CoTBench, a 1,079-image medical VQA benchmark with expert-calibrated step-by-step CoT annotations and four CoT-centric metrics (correctness, efficiency, impact, consistency), used to evaluate 15+ general and medical MLLMs.

Problem

Existing medical image VQA benchmarks score only final-answer accuracy and lack annotated intermediate reasoning, so the quality, cost, and effect of chain-of-thought reasoning in medical MLLMs cannot be assessed.

Approach

A medical VQA benchmark where each question carries clinician-reviewed four-step reasoning annotations (modality, key features, conclusion, additional analysis), paired with metrics that measure CoT step correctness (P/R/F1), efficiency (correct steps per time and latency ratio), accuracy impact of CoT versus direct answering, and path consistency via normalized LCS.

Key results

• Gemini 2.5 Pro achieves the best CoT correctness (F1 66.07%) and direct-to-CoT accuracy of 60.10% with CoT.
• CoT prompting lowers accuracy for most open-source and medical models (e.g., Lingshu-7B drops 7.92 points), while Qwen3-VL-Thinking-8B gains +4.50.
• Qwen3-VL-Instruct-8B's latency rises by 93.94x under CoT, while Qwen3-VL-Thinking-30B adds only 1.15x.

Surprising findings

• CoT prompting reduces accuracy for most models, and for many medical MLLMs the drop is large.
• GPT-5 has high precision but low recall because it often skips step-by-step reasoning even when prompted for CoT.

Limitations

• Annotations are generated mainly by GPT-4o and Gemini 2.5 Pro and then calibrated, so model-generated bias may persist.
• Efficiency and latency measurements depend on the hardware and API setup.

Experimental setup

Datasets: VQA-RAD SLAKE Quilt-VQA OphthalVQA IDRiD BUSI ISIC 2018 VQA-Med-2019 FetReg BACH

Benchmarks: M3CoTBench

Metrics: F1 precision recall accuracy reasoning impact latency ratio path consistency correct steps per unit time

Baselines: LLaVA-CoT InternVL3.5 Qwen3-VL GPT-4.1 GPT-5 Gemini 2.5 Pro Claude-Sonnet-4.5 LLaVA-Med HuatuoGPT-Vision HealthGPT Lingshu MedGemma

Builds on: Visual-CoT MME-CoT BiomedCLIP

Closest papers by meaning

• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.91)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.90)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.90)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.89)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.89)
• Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.89)