Research map / Large language models / Agents, code and math
MedRAX: Medical Reasoning Agent for Chest X-ray
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedRAX, a training-free ReAct agent that orchestrates CXR analysis tools and a multimodal LLM (GPT-4o), plus ChestAgentBench, a 2,500-question six-choice benchmark built from Eurorad cases.
Problem
Chest X-ray interpretation tools operate in isolation and existing medical VQA benchmarks test only simple single-step reasoning, so complex multi-step CXR reasoning is poorly served and poorly evaluated.
Approach
A LLM-driven ReAct agent that dynamically calls a toolbox of specialized CXR models (VQA, segmentation, grounding, report generation, classification, generation) without any additional training, paired with a new GPT-4o-generated six-choice benchmark from expert-curated Eurorad cases.
Key results
- On ChestAgentBench, MedRAX reaches 63.1% overall accuracy versus 57.9% for Llama-3.2-90B and 56.4% for GPT-4o.
- On SLAKE chest X-ray VQA (114 samples), MedRAX achieves 90.35% accuracy versus 85.1% for M4CXR.
- On MIMIC-CXR single-image report generation, MedRAX gets mF1-14 of 79.1% versus 60.6% for M4CXR, but lower macro-averaged MF1-14 (34.2% vs 40.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MedRAX's tool-orchestrated agent instead of the strongest listed baseline | CXR interpretation models · ChestAgentBench — Overall | accuracy: 57.9 -> 63.1 | Helped |
| Use MedRAX's tool-orchestrated agent instead of the strongest listed baseline | CXR interpretation models · CheXbench — Overall | accuracy: 64.7 -> 68.1 | Helped |
| Use MedRAX's tool-orchestrated agent instead of GPT-4o | CXR VQA models · CheXbench — Rad-Restruct | accuracy: 53.9 -> 68.7 | Helped |
| Use MedRAX's tool-orchestrated agent instead of GPT-4o | CXR VQA models · CheXbench — SLAKE | accuracy: 85.4 -> 82.9 | Hurt |
| Use MedRAX's tool-orchestrated agent instead of CheXagent | CXR image-text reasoning models · CheXbench — Fine-Grained Reasoning | accuracy: 59.0 -> 52.6 | Hurt |
| Use MedRAX instead of M4CXR | radiology report-generation models · MIMIC-CXR test | mF1-14: 60.6 -> 79.1 | Helped |
| Use MedRAX instead of M4CXR | radiology report-generation models · MIMIC-CXR test | mF1-5: 61.8 -> 64.9 | Helped |
| Use MedRAX instead of M4CXR | radiology report-generation models · MIMIC-CXR test | MF1-14: 40.0 -> 34.2 | Hurt |
| Use MedRAX instead of Med-PaLM M 84B | radiology report-generation models · MIMIC-CXR test | MF1-5: 51.6 -> 48.2 | Hurt |
| Use MedRAX instead of M4CXR | medical VQA models · SLAKE chest X-ray VQA | accuracy: 85.1 -> 90.35 | Helped |
| Use MedRAX instead of M4CXR | medical VQA models · SLAKE chest X-ray VQA | recall: 86.0 -> 91.23 | Helped |
Surprising findings
- General-purpose VLMs (GPT-4o, Llama-3.2-90B) outperform medical-specialized models (LLaVA-Med, CheXagent) across ChestAgentBench categories.
- On image-text reasoning in CheXbench, the best model (CheXagent, 59.0%) is nearly at random chance (50%).
Limitations
- Struggles to resolve contradictory outputs from classification and segmentation tools in fine-grained visual tasks.
- Computational overhead from running multiple tools slows response times versus end-to-end models.
- Lacks robust uncertainty quantification.
Experimental setup
Datasets: Eurorad chest imaging cases CheXbench (Rad-Restruct, SLAKE, OpenI) MIMIC-CXR CheXpert Plus ChestX-Det SLAKE
Benchmarks: ChestAgentBench CheXbench MIMIC-CXR report generation SLAKE VQA
Metrics: accuracy micro F1 (mF1-14, mF1-5) macro F1 (MF1-14, MF1-5) recall
Baselines: LLaVA-Med CheXagent Llama-3.2-90B Vision GPT-4o RadFM MAIRA-1 LLaVA-Rad Med-PaLM M 84B M4CXR
Builds on: ReAct LangGraph LLaVA-Med CheXagent M4CXR
Closest papers by meaning
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.88)
- AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.87)
- From Complexity to Clarity: Transforming Chest X-ray Reports with Chained Prompting (Student Abstract) (AAAI 2025 · similarity 0.86)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.85)
- MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.85)
- MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.