Research map / Vision-language and multimodal / Multimodal LLMs
VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VILA-M3, a medical VLM that learns to trigger domain expert models (VISTA3D, BRATS, TorchXRayVision) and incorporates their feedback via an extra expert-guided instruction fine-tuning stage.
Problem
General-purpose and existing medical VLMs lack the fine-grained precision needed for clinical tasks such as segmentation, classification, report generation and VQA, and are evaluated on static datasets with unreliable LLM judges.
Approach
Adds a fourth training stage, expert-guided instruction fine-tuning, where the VLM learns to emit keyword/argument triggers for on-demand expert segmentation and classification models and consumes their outputs as feedback in the conversation, combined with a balanced medical data mixture.
Key results
- VILA-M3-40B reaches 64.3 total average across medical benchmarks versus 55.7 for Med-Gemini (1.5T) and 58.6 for task-specific SOTA.
- With expert feedback, VILA-M3 classification F1 on CheXpert averages about 60.5-61.5 across sizes, versus 48.4 for Med-Gemini.
- VILA-M3 degrades on general VLM benchmarks (e.g., 3B drops ~7%, 8B ~11%, 40B ~23%), indicating trade-off with general capability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add expert-model feedback to VILA-M3-3B (ablation) | VILA-M3-3B · ChestX-ray14 classification | F1: 48.4 -> 51.3 | Helped |
| add expert-model feedback to VILA-M3-13B (ablation) | VILA-M3-13B · CheXpert classification | F1: 55.8 -> 61.5 | Helped |
| add expert-model feedback to VILA-M3-3B (ablation) | VILA-M3-3B · MIMIC-CXR report generation | ROUGE: 31.4 -> 31.7 | Helped |
| add expert-model feedback to VILA-M3-8B (ablation) | VILA-M3-8B · MIMIC-CXR report generation | BLEU-4: 21.4 -> 21.2 | Hurt |
| add expert-model feedback to VILA-M3-40B (ablation) | VILA-M3-40B · MIMIC-CXR report generation | BLEU-4: 21.6 -> 21.6 | No clear effect |
| balance the healthcare training data (ablation) | VILA-M3 · medical benchmark metrics | average across metrics: ~4% | Helped |
| train VILA-M3 for 3 epochs instead of 1 or 2 (ablation) | VILA-M3 model variants · medical benchmarks | performance | Hurt |
| fine-tune VILA-1.5 as VILA-M3 with expert-guided IFT | VILA-M3 · traditional VLM benchmarks average | average: 75.5 -> 52.2 | Hurt |
Surprising findings
- GPT-4o refused to answer many classification prompts when expert-model information was appended, making quantitative comparison unfair.
- Expert-guided IFT for 40B (Yi-34B backbone) caused a large drop on general benchmarks such as SQA-I (to 0.0).
Limitations
- General-benchmark performance degrades substantially after expert-guided IFT, especially for 40B.
- Off-the-shelf VLM judges such as GPT-4o are judged unreliable, and the paper gives no robust open evaluation of generative outputs beyond GREEN.
- Relies on a fixed set of expert models; the paper notes agentic multi-expert design is not yet implemented.
Experimental setup
Datasets: SLAKE PathVQA RadVQA MIMIC-CXR MIMIC-Diff-VQA ChestX-ray14 CheXpert BRATS 2018 MSD TotalSegmentatorV2 USMLE
Benchmarks: RadVQA MIMIC-VQA SLAKE PathVQA MIMIC-CXR report generation ChestX-ray14 CheXpert VQAv2 GQA VizWiz SQA-I VQA-T
Metrics: accuracy F1 BLEU-4 ROUGE GREEN
Baselines: Med-Gemini GPT-4o Llava-Med Llava-Rad DCL TorchXRayVision VILA1.5
Builds on: VILA LLaVA Vicuna Llama-3 Yi
Closest papers by meaning
- GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.90)
- Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.89)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.88)
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.88)
- HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.88)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.