Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Neural-MedBench, a compact 120-case (200-task) neurology benchmark with MRI, EHR, and notes, and shows VLMs fail mostly on clinical reasoning versus humans under a hybrid LLM-grader plus clinician evaluation.

Problem

Medical VLM benchmarks emphasize classification accuracy and may overstate clinical reasoning ability; the paper targets deep multimodal diagnostic reasoning in neurology.

Approach

Proposes a Two-Axis Evaluation Framework (breadth vs. depth) and a compact, expert-curated neurology benchmark with multi-task diagnostic reasoning, scored by a clinician-calibrated LLM grader alongside BERTScore and error taxonomy.

Key results

• MedGemma-27B-it reaches 30.0% pass@1 on Direct Diagnosis tasks versus 40.0% for senior physicians.
• Gemini 2.5-Pro achieves the highest pass@5 of 50.0% on Direct Diagnosis, and no model consistently exceeds 50%.
• Of 100 analyzed incorrect responses, 51% were reasoning failures versus 27% perceptual failures.

Surprising findings

• Medical-specialized models do not consistently outperform generalist models; Gemini 2.5-Pro leads pass@5 while MedGemma leads pass@1 on direct diagnosis.
• Medical students score lower than VLMs on pass@1 for direct diagnosis (3.3% vs ~30%) but are better in multi-turn dialogue.

Limitations

• Small benchmark (120 cases, 200 tasks) limits statistical claims.
• Grader validation relies on GPT-4o, which is also a benchmarked model; human baseline has only 10 participants.
• Condition distribution biased toward imaging-visible diseases; no multi-center domain shift coverage.

Experimental setup

Datasets: ADNI OASIS Radiopaedia JAMA Neurology case reports

Benchmarks: Neural-MedBench

Metrics: pass@1 pass@5 BERTScore error taxonomy distribution

Baselines: GPT-4o Claude 4.0 Sonnet Gemini 2.5-Pro MedGemma Lingshu LLaVA-Med Huatuo-GPT-Vision RadFM Med-Flamingo

Builds on: OSCE-style clinical assessment BERTScore

Closest papers by meaning

• Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.90)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.90)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.90)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.89)
• OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.89)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.89)