Research map / Vision-language and multimodal / Multimodal LLMs

DDx-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DDx-TRACE, a physician-adjudicated neuroradiology benchmark where VLMs request imaging hidden-evidence step by step, update differentials, and stop with a localized diagnosis, scored on process and endpoint metrics.

Problem

Medical VLM benchmarks reveal all evidence upfront and score only the final answer, hiding whether models acquire essential evidence, interpret images, update beliefs, and stop appropriately during a diagnostic workup.

Approach

Reframes multimodal diagnosis as a turn-based hidden-evidence workup: models issue free-form imaging requests resolved against a hidden inventory of physician-annotated exam bundles, output a four-item probabilistic differential each turn, and are scored on essential-evidence recall, order concordance, request burden, stopping behavior, and confidence alignment alongside endpoint accuracy.

Key results

• Best frontier model Gemini 3 Flash reaches a clinically supported diagnosis in only 30.9% of cases, and all medical/radiology-adapted VLMs have single-digit reached rates (among the 12 models in the efficiency table, 8 are single-digit).
• Gemini 3.1 Pro achieves the best endpoint diagnosis score (S_dx=0.546) and S_ddx=0.644, while Gemini 3 Flash leads process metrics (S_ER=0.767, S_order=0.619, S_conf=0.531), showing endpoint and process rankings diverge.
• Providing oracle textual findings raises average S_dx from 0.39 to 0.58 and S_ER to 0.73, while history-only drops S_dx to 0.26, indicating visual evidence extraction is a major bottleneck.

Surprising findings

• Models with the highest endpoint accuracy do not necessarily have the best workup quality; GPT-5.4 Mini has low unmatched-request rates but weak essential-evidence recall.
• Model performance does not drop markedly on physician-rated rarer or harder cases, suggesting the bottleneck is process (planning and acquisition) rather than case difficulty.

Limitations

• Neuroradiology only, based on retrospective EuroRad teaching cases with key images rather than full PACS studies.
• Hidden finite inventory means clinically reasonable but unavailable requests are counted as unmatched.
• Inter-annotator agreement is modest (kappa 0.19-0.36 for several label families) and LLM-drafted templates needed heavy physician correction.

Experimental setup

Datasets: EuroRad teaching cases

Benchmarks: DDx-TRACE

Metrics: S_dx S_loc S_ddx S_ER S_order B_opt B_unm S_conf S_traj T_guess

Baselines: GPT-5.4 GPT-5.4 Mini Gemini 3 Flash Gemini 3.1 Pro Claude Sonnet 4.6 Qwen3.5-35B-A3B Qwen3.5-27B Qwen3.5-9B Qwen3.5-4B Gemma 3 27B MedGemma 27B MedGemma 1.5 4B

Builds on: EuroRad teaching cases MedXpertQA VQA-RAD

Closest papers by meaning

• DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis? (NeurIPS 2025 · similarity 0.86)
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis (NeurIPS 2025 · similarity 0.86)
• MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis (AAAI 2026 · similarity 0.86)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.86)
• MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.86)
• CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models (NeurIPS 2026 · similarity 0.85)