Research map / Vision-language and multimodal / Multimodal LLMs
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BioMed-VITAL, a data-centric pipeline that generates biomedical visual instruction data with GPT-4V using clinician-selected demonstrations, then selects high-quality samples with a BiomedCLIP-based rating model distilled from clinician and GPT-4V preferences, and instruction-tunes LLaVA-13B.
Problem
Automatically generated biomedical visual instruction-tuning data is not explicitly aligned with clinician expertise, and clinician preference annotations are scarce while strong generators like GPT-4V are proprietary.
Approach
Inject clinician preferences at two stages: as few-shot demonstrations for GPT-4V data generation, and as a mixed human-plus-model preference signal (with a weighting factor) used to train a pairwise rating model that filters and ranks generated data.
Key results
- Full BioMed-VITAL improves overall open visual chat score to 68.63 vs 57.92 for LLaVA-Med (about 18.5% relative gain).
- Win rates against LLaVA-Med reach up to 81.73% on biomedical VQA benchmarks as judged by GPT-4V.
- The 150K-trained BioMed-VITAL (13B) reaches 69.72 open-ended recall on VQA-RAD and 91.69 on SLAKE open-set, the best among LLaVA-based models reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use clinician preference alignment in both data-generation and selection stages | BioMed-VITAL · Open-ended medical visual chat (overall) | GPT-4V score: 57.92 -> 68.63 | Helped |
| Use clinician preference alignment in both stages rather than neither (ablation) | BioMed-VITAL · Open-ended medical visual chat (overall) | GPT-4V score: 64.15 -> 68.63 | Helped |
| Add preference distillation in selection to demonstrations-only alignment (ablation) | BioMed-VITAL · Open-ended medical visual chat (overall) | GPT-4V score: 65.66 -> 68.63 | Helped |
| Add clinician-selected demonstrations in generation to selection-only alignment (ablation) | BioMed-VITAL · Open-ended medical visual chat (overall) | GPT-4V score: 66.30 -> 68.63 | Helped |
| Increase training data from 60K to 150K (ablation) | BioMed-VITAL (13b) · VQA-RAD test | open-question recall: 64.88 -> 69.72 | Helped |
| Increase training data from 60K to 150K (ablation) | BioMed-VITAL (13b) · SLAKE test | open-question recall: 87.82 -> 91.69 | Helped |
| Increase training data from 60K to 150K (ablation) | BioMed-VITAL (13b) · PathVQA test | open-question recall: 39.71 -> 39.89 | Helped |
| Increase selected training data from top 10% to top 80% (ablation) | BioMed-VITAL · Open-ended medical visual chat (overall) | GPT-4V score: 63.06 -> 68.28 | Helped |
| Set preference-mixture weight to 400 (ablation) | BiomedCLIP selection model · Clinician-annotated preference test set | ACC (%): 63.64 -> 66.72 | Helped |
| Use clinician preference alignment in both generation and selection instead of either stage alone (ablation) | BioMed-VITAL · Biomedical VQA benchmarks | GPT-4V win rate: win rate up to 81.73% | Helped |
| Use generation-stage clinician alignment only rather than selection-stage alignment only (ablation) | BioMed-VITAL ablations · Biomedical VQA benchmarks | GPT-4V win rate | Helped |
Surprising findings
- Training on only the top 10% of selected data already surpasses LLaVA-Med trained on the full dataset on visual chat.
- The selection model's alignment with clinician preference reportedly exceeds that of GPT-4 as a judge.
Limitations
- Modality imbalance in PMC-15M subset may bias instruction tuning.
- Noise, low-quality data and hallucination from GPT-4V generation may remain after selection.
- Only tested on the PMC-15M data; not applied to other resources due to compute limits.
Experimental setup
Datasets: PMC-15M VQA-RAD SLAKE PathVQA LLaVA-Med instruction data
Benchmarks: LLaVA-Med open-ended visual chat (193 QA pairs) VQA-RAD SLAKE PathVQA
Metrics: GPT-4V relative score (1-10) recall for open questions accuracy for closed questions win rate ACC AUC MR MAP F1@K Precision@K
Baselines: LLaVA-Med LLaVA MMQ Prefix T. Medical LM PubMedCLIP BiomedCLIP M2I2 MUMC M3AE CoQAH PMC-CLIP
Builds on: LLaVA LLaVA-Med BiomedCLIP PMC-15M
Closest papers by meaning
- MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants (NeurIPS 2025 · similarity 0.88)
- QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training (NeurIPS 2025 · similarity 0.86)
- OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning (CVPR 2026 · similarity 0.85)
- Detecting Any instruction-to-answer interaction relationship:Universal Instruction-to-Answer Navigator for Med-VQA (ICML 2024 · similarity 0.84)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.84)
- OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.