Research map / Vision-language and multimodal / Human and embodied modeling
RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAD, which retrieves and LLM-refines disease-specific guidelines offline and uses them via a guideline-enhanced contrastive loss and dual transformer decoder to improve multimodal clinical diagnosis and interpretability.
Problem
Multimodal clinical diagnosis models rely on implicit knowledge in parameters and do not explicitly inject task-specific clinical guideline knowledge, making decisions hard to trust or interpret.
Approach
Offline retrieval (MedCPT) of disease-specific documents from Wiki, PubMed, guidelines and books, LLM-summarized into per-disease guidelines that serve as prototypes for a supervised contrastive loss on image/text features and as queries in a dual decoder fusion module for multi-label prediction.
Key results
- RAD improves average score over second-best method by 3.09 points on MIMIC-ICD53 (57.28 avg), 2.24 on FairVLMed, 2.49 on SkinCAP, and 2.45 on NACC.
- Guideline recall rises from 24.76% to 65.62% with RAD on MIMIC-ICD53.
- Zero-shot lesion grounding mIoU on ChestX-Det improves from 15.98 to 19.72 (disease-averaged).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply RAD instead of KAD | MIMIC-ICD53 diagnosis · MIMIC-ICD53 | Avg: 54.19 -> 57.28 | Helped |
| apply RAD instead of HEALNet | FairVLMed diagnosis · FairVLMed | Avg: 84.39 -> 86.63 | Helped |
| apply RAD instead of KAD | SkinCAP diagnosis · SkinCAP | Avg: 86.15 -> 88.64 | Helped |
| apply RAD instead of KAD | NACC diagnosis · NACC | Avg: 55.77 -> 58.12 | Helped |
| add both GECL branches and the dual decoder (ablation) | RAD · MIMIC-ICD53 | Avg: 53.35 -> 57.28 | Helped |
| add the dual decoder rather than an MLP (ablation) | RAD with both GECL branches · MIMIC-ICD53 | Avg: 56.26 -> 57.28 | Helped |
| add both GECL branches (ablation) | RAD with dual decoder · MIMIC-ICD53 | Avg: 55.91 -> 57.28 | Helped |
| use text GECL rather than vision GECL alone (ablation) | RAD with MLP decoder · MIMIC-ICD53 | Avg: 54.79 -> 55.52 | Helped |
| add the vision GECL branch to text GECL (ablation) | RAD with MLP decoder · MIMIC-ICD53 | Avg: 55.52 -> 56.26 | Helped |
| inject RAD guidelines | multimodal model visual grounding · MIMIC-ICD53 visual grounding | mIoU Avg-D: 15.98 -> 19.72 | Helped |
| inject RAD guidelines | multimodal model textual indicator attention · MIMIC-ICD53 textual explainability | Guideline Recall Total: 24.76 -> 65.62 | Helped |
Surprising findings
- Sample-wise accuracy (ACC-S) is much lower than ACC across all methods, especially with large label spaces.
- Conventional models show very low guideline recall (<10%) on ALT and AST indicators.
Limitations
- Static retrieval knowledge corpus that needs manual updates as guidelines change.
- Guidelines and indicators were manually verified, so the pipeline is not fully automated.
- MIMIC-ICD53 can only be released on PhysioNet, limiting open access.
Experimental setup
Datasets: MIMIC-CXR MIMIC-IV MIMIC-ICD53 Harvard-FairVLMed SkinCAP NACC ChestX-Det Wikipedia PubMed Clinical Guidelines dataset MedOmniKB
Benchmarks: MIMIC-ICD53 Harvard-FairVLMed SkinCAP NACC ChestX-Det
Metrics: F1 Precision Recall AUC mAP Acc Acc-S Guideline Recall mIoU
Baselines: MedFuse BiomedCLIP KAD DrFuse HEALNet Qwen2.5-VL-7B HuatuoGPT-Vision-7B Lingshu-7B
Builds on: MedCPT CLIP SupCon ClinicalBERT
Closest papers by meaning
- MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis (CVPR 2026 · similarity 0.88)
- MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.88)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.88)
- Medical thinking with multiple images (ICLR 2026 · similarity 0.87)
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.87)
- 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.