Research map / Vision-language and multimodal / Multimodal LLMs

CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CheXStruct, an automated segmentation-based pipeline that derives structured diagnostic reasoning steps from chest X-rays, and CXReasonBench, a multi-path, multi-stage benchmark (18,988 QA pairs, 12 tasks, 1,200 MIMIC-CXR-JPG cases) evaluating 12 LVLMs.

Problem

Existing chest X-ray VQA benchmarks score only final diagnostic answers, giving little insight into whether LVLMs perform clinically valid intermediate reasoning steps such as anatomical grounding, measurement, and threshold application.

Approach

Automatically derive reference reasoning chains (segmentation → landmarks → measurements → indices → thresholds) from chest X-ray images using CXAS segmentation and rule-based clinical criteria with QC filtering, then evaluate models via a multi-stage Path 1 (direct reasoning reconstruction) and Path 2 (guided reasoning followed by re-evaluation on a new case).

Key results

• Gemini-2.5-Pro is the strongest Path 1 model, reaching 17.03% final-stage completion and average depth 1.96 of 4, rarely completing the full reasoning chain.
• In Path 2 guided reasoning, Gemini-2.5-Pro achieves 55.81% completion, while most models struggle to transfer guidance to re-evaluated new cases (e.g., Gemini-2.5-Pro 0.0% alignment in re-evaluated Path 1).
• Stage 2 anatomical grounding varies by task: inclusion 89% and carina angle 79%, versus inspiration level 47% and tracheal deviation 48%.

Measured effects reported in the paper

Surprising findings

• Medical-specialized models (HealthGPT, RadVLM) do better on recognition-type tasks but worse on measurement-type tasks, while most general-domain models show the opposite trend.
• Most models show lower measurement consistency under Path 1 than Path 2, suggesting heuristic approximation rather than computation.

Limitations

• Only 12 structurally inferable tasks; pixel-level pathologies such as opacities or effusion are excluded.
• Relies on CXAS segmentation quality; QC filtering removes some images (e.g., 3.12% of PA/AP images globally).
• Some alignment metrics are less definitive for tasks without standardized thresholds.

Experimental setup

Datasets: MIMIC-CXR-JPG PAX-Ray++ Chest ImaGenome CheXmask

Benchmarks: CXReasonBench

Metrics: completion rate average reasoning depth decision alignment measurement consistency

Baselines: Gemini-2.5-Flash GPT-4.1 Pixtral-Large Llama-3.2-90B-Vision Qwen2.5-VL-72B Pixtral 12B Qwen2.5-VL-7B MedGemma 27B HealthGPT-L14 RadVLM MedGemma 4B

Builds on: CXAS MIMIC-CXR-JPG CheXmask Chest ImaGenome Wilson score

Closest papers by meaning

• AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.91)
• BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.90)
• Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models (NeurIPS 2026 · similarity 0.90)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.89)
• LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (ICLR 2024 · similarity 0.88)
• MedRAX: Medical Reasoning Agent for Chest X-ray (ICML 2025 · similarity 0.88)