Research map / Vision-language and multimodal / Multimodal LLMs
AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AOR, an anatomy-centric multimodal reasoning framework for chest X-ray MLMMs, plus the AOR-Instruction dataset (anatomical-ontology CoT VQA and region-sentence report data), trained in three stages on LLaVA-1.5 with CLIP-ViT-L/14.
Problem
Medical large multimodal models for chest X-ray interpretation lack region-level understanding/interaction and rely on single-step, black-box reasoning, limiting accuracy and interpretability.
Approach
Use anatomical ontologies (part-whole hierarchy, attribute causality, object-attribute restrictions) written with expert physicians to build region-grounded chain-of-thought answers, and feed multi-scale RoI features of referenced anatomical regions into the LLM during reasoning, with a three-stage recognition, grounding, instruction-tuning curriculum.
Key results
- AOR-t reaches 80.48 verify, 71.96 choose and 65.05 query on MIMIC-CXR-VQA, outperforming the second-best method by an average of 6.98%.
- Full three-stage training yields 98.58% referring accuracy and 90.40% R@0.7 grounding recall.
- AOR outperforms prior methods on full-image and region report generation on MIMIC-CXR, with an average 5.27% gain over the second-best MLMM reported in the introduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AOR anatomy-centered reasoning instead of the strongest baseline | AOR-t · MIMIC-CXR-VQA | choose accuracy: 58.70 -> 71.96 | Helped |
| Use AOR anatomy-centered reasoning instead of the strongest baseline | AOR-t · MIMIC-CXR-VQA | query F1 (micro): 60.70 -> 65.05 | Helped |
| Use AOR anatomy-centered reasoning instead of the strongest baseline | AOR-t · VQA-RAD | closed accuracy: 61.64 -> 63.01 | Helped |
| Use AOR anatomy-centered reasoning instead of the strongest baseline | AOR-t · CheXpert | open recall: 42.08 -> 53.85 | Helped |
| Train with textual and visual question prompts instead of textual questions only | AOR · MIMIC-CXR-VQA | verify accuracy: 80.48 -> 80.68 | Helped |
| Train with textual and visual question prompts instead of textual questions only | AOR · VQA-RAD | closed accuracy: 63.01 -> 57.53 | Hurt |
| Train with textual and visual question prompts instead of textual questions only | AOR · CheXpert | closed accuracy: 71.58 -> 74.06 | Helped |
| Add Stage 2 anatomical region grounding (ablation) | AOR without Stage 2 · MIMIC-CXR-VQA | grounding R@0.7: 89.40 -> 90.40 | Helped |
| Add Stage 1 anatomical region recognition (ablation) | AOR without Stage 1 or Stage 2 · MIMIC-CXR-VQA | referring accuracy: 98.03 -> 98.30 | Helped |
| Add Stage 1 anatomical region recognition (ablation) | AOR without Stage 1 or Stage 2 · MIMIC-CXR-VQA | choose accuracy: 67.96 -> 70.05 | Helped |
Surprising findings
- Performance remains nearly unchanged for bbox shifts up to 0.2 of image size, but drops when shifts exceed 0.3.
Limitations
- Region-level reasoning degrades when user-provided bounding boxes are shifted substantially, causing region misclassification.
- CoT templates depend on expert physician design and review, limiting scalability.
- Evaluation on report generation covers only 500 MIMIC-CXR test images and sampled regions.
Experimental setup
Datasets: MIMIC-CXR-VQA MIMIC-CXR Chest ImaGenome RadGraph
Benchmarks: MIMIC-CXR-VQA VQA-RAD CheXpert MIMIC-CXR
Metrics: accuracy F1 (micro) recall ROUGE-L BERTScore F1CheXbert R@0.3 R@0.5 R@0.7
Baselines: LLaVA LLaVA-1.5 GPT4RoI VoCoT LLaVA-Med Med-Flamingo XrayGPT CheXagent
Builds on: LLaVA-1.5 GPT4RoI Chest ImaGenome MIMIC-CXR-VQA RadGraph
Closest papers by meaning
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.91)
- Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models (NeurIPS 2026 · similarity 0.88)
- LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (ICLR 2024 · similarity 0.88)
- Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.88)
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.87)
- MedRAX: Medical Reasoning Agent for Chest X-ray (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.