Research map / Vision-language and multimodal / Multimodal LLMs
DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DrVD-Bench, a hierarchical multimodal benchmark (7,789 QA pairs over CT, MRI, ultrasound, radiography, pathology) that tests whether VLMs reason stepwise like clinicians across visual evidence, reasoning trajectory, and report generation.
Problem
Existing medical VLM benchmarks do not systematically test whether models follow clinician-like stepwise visual reasoning or rely on shortcuts and memorized patterns.
Approach
A five-level clinically motivated task hierarchy (image quality, basic info, anatomy, lesion, clinical interpretation) with organ and lesion erasure tasks to test visual grounding, plus Joint, Independent, and Multi-turn QA formats to probe stepwise reasoning trajectories.
Key results
- GPT-o3 reaches 86% on basic information but drops to 66% on organ-level and 41% on lesion-level CT tasks.
- Gemini 2.5 Pro achieves the best overall visual evidence accuracy (69.3%), while HuatuoGPT-Vision-34B (66%) is second with fewer than half the parameters of top models.
- Models perform best on Joint QA and worst on Multi-turn QA in the reasoning trajectory module.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase reasoning complexity from basic information to anatomy and lesion recognition | VLMs · CT tasks | accuracy: 86% -> 66% -> 41% | Hurt |
| Use Multi-turn QA instead of Joint QA | VLMs · Reasoning Trajectory Assessment, diagnosis | accuracy: 55 -> 35 | Hurt |
| Use Joint QA instead of Multi-turn QA | VLMs · Reasoning Trajectory Assessment | accuracy | Helped |
| Require lesion evidence through CT lesion-erasure identification | GPT-o3 · CT lesion erasure identification | accuracy: 19% | Hurt |
| Require lesion evidence through CT lesion-erasure identification | Gemini 2.5 Pro · CT lesion erasure identification | accuracy: 16% | Hurt |
| Use diagnosis prediction rather than lesion-level identification | GPT-o3 · CT tasks | accuracy: 28% -> 52% | Helped |
| Use diagnosis prediction rather than lesion-level identification | Gemini 2.5 Pro · CT tasks | accuracy: 28% -> 71% | Helped |
| Evaluate single-image pathology rather than ultrasound | VLMs · Visual evidence comprehension | accuracy: Pathology 81.6; Ultrasound 59.4 | Mixed |
| Use medical-specific optimization rather than general-purpose modeling | HuaTuoGPT-Vision-34B · DrVD-Bench visual evidence comprehension, overall | accuracy: 66 | Helped |
Surprising findings
- Several models score below random on CT lesion erasure detection (GPT-o3 19%, Gemini 2.5 Pro 16%) yet reach much higher diagnosis accuracy, indicating overdiagnosis without visual grounding.
- Multi-turn QA, which mirrors stepwise clinical reasoning, yields the weakest performance across models.
Limitations
- Controlled evaluation lacking real-world patient context such as clinical notes and disease progression.
- Structured task design may not capture the non-linearity and variability of real clinical decision-making.
- Patient re-identification privacy risks remain despite de-identification.
Experimental setup
Datasets: MIMIC-CXR v4 MURA VinDr-SpineXR SA-Med2D-20M RSNA-Pneumonia AMOS 2022 DeepLesion CT-RATE PubMedVision LiTS KiTS 2021 BraTS 2020
Benchmarks: DrVD-Bench
Metrics: accuracy normalized BERTScore
Baselines: GPT-4o GPT-o1 GPT-o3 Gemini 2.5 Pro Claude 3.7 Sonnet Grok-3 Doubao-1.5-VisionPro Qwen2.5-VL HuatuoGPT-Vision-34B LLaVA-Med-7B
Builds on: LLaVA BERTScore PubMedVision PathMMU
Closest papers by meaning
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.89)
- OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.89)
- Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.89)
- Medical thinking with multiple images (ICLR 2026 · similarity 0.89)
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.89)
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.