Research map / Vision-language and multimodal / Multimodal LLMs
MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMed-RAG, a multimodal RAG system for Med-LVLMs combining domain-aware retrieval, similarity-gap adaptive context selection, and RAG-based DPO fine-tuning to improve factual accuracy across radiology, ophthalmology, and pathology.
Problem
Med-LVLMs produce non-factual responses; existing medical RAG methods are dataset-specific and cause cross-modal and overall misalignment when retrieved contexts are added.
Approach
Routes each medical image to a domain-specific CLIP-style retriever via a fine-tuned BiomedCLIP domain classifier, truncates retrieved contexts at large similarity-ratio drops, and fine-tunes the Med-LVLM with DPO on preference pairs built from noised-image negatives and with/without-retrieval correctness to align the model with both image and retrieved knowledge.
Key results
- On medical VQA, MMed-RAG improves LLaVA-Med-1.5 by 18.5% on average and beats the best RAG baseline (RULE) by 2.8%.
- On report generation, MMed-RAG improves the original Med-LVLM by 69.1% and outperforms RAG baselines by 16.1%.
- Ablation on IU-Xray VQA: domain-aware retrieval raises accuracy from 68.99 to 77.12, context selection to 79.56, and RAG-PT to 85.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add domain-aware retrieval (ablation) | LLaVA-Med-1.5 · IU-Xray VQA | score: 68.99 -> 77.12 | Helped |
| add domain-aware retrieval (ablation) | LLaVA-Med-1.5 · IU-Xray report generation | score: 10.04 -> 13.23 | Helped |
| add domain-aware retrieval (ablation) | LLaVA-Med-1.5 · Harvard-FairVLMed VQA | score: 66.63 -> 72.69 | Helped |
| add domain-aware retrieval (ablation) | LLaVA-Med-1.5 · Harvard-FairVLMed report generation | score: 13.41 -> 15.89 | Helped |
| add adaptive retrieved context selection after DR (ablation) | LLaVA-Med-1.5 + DR · IU-Xray VQA | score: 77.12 -> 79.56 | Helped |
| add adaptive retrieved context selection after DR (ablation) | LLaVA-Med-1.5 + DR · IU-Xray report generation | score: 13.23 -> 17.92 | Helped |
| add adaptive retrieved context selection after DR (ablation) | LLaVA-Med-1.5 + DR · Harvard-FairVLMed VQA | score: 72.69 -> 75.74 | Helped |
| add adaptive retrieved context selection after DR (ablation) | LLaVA-Med-1.5 + DR · Harvard-FairVLMed report generation | score: 15.89 -> 17.22 | Helped |
| add RAG-based preference fine-tuning after DR and RCS (ablation) | LLaVA-Med-1.5 + DR + RCS · IU-Xray VQA | score: 79.56 -> 85.80 | Helped |
| add RAG-based preference fine-tuning after DR and RCS (ablation) | LLaVA-Med-1.5 + DR + RCS · IU-Xray report generation | score: 17.92 -> 29.80 | Helped |
| add RAG-based preference fine-tuning after DR and RCS (ablation) | LLaVA-Med-1.5 + DR + RCS · Harvard-FairVLMed VQA | score: 75.74 -> 87.18 | Helped |
| add RAG-based preference fine-tuning after DR and RCS (ablation) | LLaVA-Med-1.5 + DR + RCS · Harvard-FairVLMed report generation | score: 17.22 -> 20.42 | Helped |
Surprising findings
- Adding retrieved reports to LLaVA-Med-1.5 led 55.08% of noisy-image cases to copy the retrieved answer, indicating the model ignores the image when retrieval is present.
- A general retriever mixing all modalities underperformed domain-specific retrievers, even with MoE (CLIP-MoE) variants.
Limitations
- Dataset-specific report-based VQA construction via GPT-4 with yes/no answers and manual filtering may limit generality.
- Domain-specific retrievers require retraining when a new modality is encountered.
- Gains rely on a fixed threshold gamma for context truncation.
Experimental setup
Datasets: MIMIC-CXR IU-Xray Harvard-FairVLMed PMC-OA Quilt-1M
Benchmarks: MIMIC-CXR IU-Xray Harvard-FairVLMed PMC-OA Quilt-1M PubMedVision
Metrics: Accuracy F1 AUROC BLEU ROUGE-L METEOR
Baselines: Greedy decoding Beam search DoLa OPERA VCD MedDr FactMM-RAG RULE Med-Flamingo MedVInT RadFM miniGPT-Med
Builds on: DPO LLaVA-Med CLIP RULE BiomedCLIP
Closest papers by meaning
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.89)
- MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization (ICML 2025 · similarity 0.88)
- GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.87)
- Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs (AAAI 2025 · similarity 0.87)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.