Research map / Vision-language and multimodal / Multimodal LLMs
MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MedTrinity-25M, a 25M-image medical dataset of image-ROI-description triplets generated automatically from unpaired images via expert grounding, RAG, and MLLM prompting, and trains LLaVA-Tri on it to reach SOTA on three medical VQA benchmarks.
Problem
Medical multimodal datasets rely on paired expert reports or captions, which limits scale, and lack multigranular annotations linking global findings to local regions.
Approach
An automated pipeline generates multigranular visual (ROI) and textual annotations for unpaired medical images by combining metadata-derived coarse captions, expert segmentation/grounding models for ROIs, retrieval-augmented medical knowledge, and a fine-tuned LLaVA (LLaVA-Medcap) prompted with a three-level hierarchical template.
Key results
- LLaVA-Tri reaches 75.0% average on VQA-RAD, 87.8% on SLAKE, and 65.3% on PathVQA after fine-tuning, reported as state of the art on these three benchmarks.
- Pretraining on MedTrinity-25M improves LLaVA-Tri by 2.0, 5.0, and 1.6 points on VQA-RAD, SLAKE, and PathVQA averages; MiniCPM-V-2.6-8B and InternVL2-8B also gain 1.6-4.4 points.
- Generated descriptions score 0.85 (expert) and 0.86 (LLM, GPT-4V) on alignment across five attributes on a 200-sample subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MedTrinity-25M alignment pretraining (ablation) | LLaVA-Tri · VQA-RAD | average accuracy: 73.0 -> 75.0 (+2.0) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | LLaVA-Tri · SLAKE | average accuracy: 82.8 -> 87.8 (+5.0) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | LLaVA-Tri · PathVQA | average accuracy: 63.7 -> 65.3 (+1.6) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | MiniCPM-V-2.6-8B · VQA-RAD | average accuracy: 67.5 -> 69.1 (+1.6) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | MiniCPM-V-2.6-8B · SLAKE | average accuracy: 68.6 -> 73.0 (+4.4) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | MiniCPM-V-2.6-8B · PathVQA | average accuracy: 60.9 -> 64.5 (+3.6) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | InternVL2-8B · VQA-RAD | average accuracy: 57.2 -> 60.4 (+3.2) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | InternVL2-8B · SLAKE | average accuracy: 69.8 -> 72.6 (+2.8) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | InternVL2-8B · PathVQA | average accuracy: 51.6 -> 55.5 (+3.9) | Helped |
| Add MedTrinity-25M alignment pretraining (ablation) | PubMedCLIP · VQA-RAD | average accuracy: 67.5 -> 70.2 (+2.7) | Helped |
Surprising findings
- Quality evaluation of descriptions was on only 200 samples, and the LLM judge (GPT-4V) scores were close to expert scores.
Limitations
- Descriptions are generated by a fine-tuned MLLM (LLaVA-Medcap) and not verified at scale by experts; quality checked on only 200 samples.
- Source-dataset ROIs depend on expert grounding models (HoverNet, SAT, DINO, CheXmask) that may fail on out-of-distribution images.
- Results on VQA-RAD/SLAKE/PathVQA are obtained by fine-tuning on each VQA dataset, and some baseline comparisons are mixed-setting.
Experimental setup
Datasets: MedTrinity-25M PMC-15M VQA-RAD SLAKE PathVQA MIMIC-CXR-JPG CheXpert DeepLesion PMC-OA LLaVA-Med CT-RATE
Benchmarks: VQA-RAD SLAKE PathVQA
Metrics: accuracy
Baselines: LLaVA-Med LLaVA GPT-4V BiomedCLIP PubMedCLIP M2I2 Q2ATransformer
Builds on: LLaVA MedRAG LLaVA-Med MedCPT
Closest papers by meaning
- MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.88)
- Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.87)
- BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · similarity 0.87)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.87)
- MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives (NeurIPS 2025 · similarity 0.86)
- GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.