Research map / Vision-language and multimodal / Multimodal LLMs
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMedPO, a clinical-aware multimodal DPO variant for medical LVLMs that builds hallucinated-text and lesion-noised-image preference pairs and weights each pair by a clinical relevance score from a multi-Med-LLM debate and visual-tool confidence.
Problem
Medical LVLMs hallucinate because of modality misalignment (favoring text priors over images); existing preference optimization data for medical VLMs is easily distinguishable and ignores clinical relevance of dispreferred samples and lesion regions.
Approach
Curate preference pairs from two sources (GPT-4o/target-model hallucinated responses and MedKLIP-guided local lesion noising of images), score each pair's clinical relevance via a multi-Med-LLM consensus debate plus visual-tool confidence, normalize the scores, and use them as weights in a DPO loss.
Key results
- MMedPO reaches 53.99 open / 73.08 closed on SLAKE and 36.36 open / 66.54 closed on VQA-RAD for LLaVA-Med v1.5, the best among compared methods.
- On report generation, MMedPO achieves BLEU 23.49 / ROUGE-L 29.52 on IU-Xray and 12.85 / 11.13 on MIMIC-CXR (without SFT), with reported gains of 61.9% and 26.0% over the best baseline.
- Applied after SFT, MMedPO improves over the SFT baseline on all four datasets, with the paper reporting an average 14.2% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MMedPO instead of FiSAO | LLaVA-Med v1.5 7B · SLAKE | Open recall: 52.69 -> 53.99 | Helped |
| apply MMedPO instead of FiSAO | LLaVA-Med v1.5 7B · SLAKE | Closed accuracy: 70.46 -> 73.08 | Helped |
| apply MMedPO instead of Self-Rewarding | LLaVA-Med v1.5 7B · VQA-RAD | Open recall: 33.29 -> 36.36 | Helped |
| apply MMedPO instead of POVID | LLaVA-Med v1.5 7B · VQA-RAD | Closed accuracy: 65.07 -> 66.54 | Helped |
| apply MMedPO instead of FiSAO | LLaVA-Med v1.5 7B · IU-Xray | BLEU: 21.06 -> 23.49 | Helped |
| apply MMedPO instead of FiSAO | LLaVA-Med v1.5 7B · MIMIC-CXR | METEOR: 7.84 -> 10.03 | Helped |
| add clinical relevance score weighting to Stage 1 (ablation) | MMedPO · IU-Xray | Average score: 10.95 -> 15.66 | Helped |
| add clinical relevance score weighting to Stage 1 (ablation) | MMedPO · SLAKE | Average score: 55.65 -> 57.62 | Helped |
| add clinical relevance score weighting to Stage 2 (ablation) | MMedPO · IU-Xray | Average score: 19.30 -> 25.00 | Helped |
| use multiple Med-LLMs rather than a single Med-LLM for clinical relevance scores (ablation) | MMedPO · VQA-RAD | Average score: 48.67 -> 51.14 | Helped |
| use multiple Med-LLMs rather than a single Med-LLM for clinical relevance scores (ablation) | MMedPO · SLAKE | Average score: 56.09 -> 57.53 | Helped |
| use localized lesion noise instead of global image noise (ablation) | MMedPO Stage 2 preference data · SLAKE | Average score: 58.88 -> 59.88 | Helped |
Surprising findings
- Single-Med-LLM vs multi-Med-LLM clinical scoring gaps are modest on report generation (e.g., BLEU-4 4.39 vs 4.40 on IU-Xray), suggesting the consensus mechanism mainly helps VQA.
- Baseline ranking is inconsistent across tasks: the original DPO is strongest on some metrics while POVID and FiSAO are strongest on report generation.
Limitations
- Reliance on GPT-4o for generating hallucinated dispreferred responses and on proprietary API for curation.
- Clinical relevance scores come from LLM judges without human clinical validation (no human evaluation reported).
- Evaluated only on chest X-ray and VQA datasets with a single base model family (LLaVA-Med).
Experimental setup
Datasets: SLAKE VQA-RAD IU-Xray MIMIC-CXR
Benchmarks: SLAKE VQA-RAD IU-Xray MIMIC-CXR
Metrics: accuracy recall BLEU ROUGE-L METEOR
Baselines: LLaVA-Med v1.5 Self-Rewarding DPO POVID SIMA FiSAO STLLaVA-Med
Builds on: DPO POVID SIMA FiSAO STLLaVA-Med
Closest papers by meaning
- Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.88)
- MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models (ICLR 2025 · similarity 0.88)
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.88)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.88)
- ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.