Research map / Vision-language and multimodal / Multimodal LLMs
MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedMax, a 1.47M-instance mixed-modal biomedical instruction-tuning dataset (including interleaved image-text generation data) and LoRA-tunes Anole-7B (Chameleon-7B) on it, reporting gains over Chameleon, LLaVA-Med, HuatuoGPT-Vision, and GPT-4o on 12 biomedical VQA tasks.
Problem
Mixed-modal foundation models struggle on biomedical vision-language data, and existing biomedical instruction-tuning datasets are small, narrow in task and domain coverage, and lack support for interleaved image-text generation and image generation.
Approach
A large, diverse biomedical instruction-tuning mix spanning VQA, captioning, image generation, visual chat, report understanding and newly curated interleaved image-text generation data, used to LoRA-tune an autoregressive mixed-modal model (Anole-7B) with a unified evaluation suite.
Key results
- MedMax-7B averages 65.5 across 12 biomedical VQA tasks, versus 39.4 for Chameleon-7B (+26.1 points) and 42.0 for GPT-4o.
- MedMax-7B reaches 99.5 on the OmniMedVQA hidden split and 91.8 on PathVQA (closed), best among compared models on those tasks.
- On multimodal generation, MedMax improves over Chameleon by 25.2% on LLM-scored text quality and 31.5% on BioMedCLIP image-image similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Chameleon-7B as Anole-7B on MedMax | Chameleon-7B · 12 VQA tasks (average) | accuracy: 39.4 -> 65.5; 26.1 percentage points | Helped |
| Fine-tune Chameleon-7B as Anole-7B on MedMax | Chameleon-7B · 12 VQA tasks (average) | accuracy: 42.0 -> 65.5; 18.3 percentage points | Helped |
| Increase the MedMax training-data subset size (ablation) | Anole-7B fine-tuned on MedMax · 12 VQA tasks (average) | accuracy | Helped |
| Add VQA task instances to the training mixture (ablation) | Anole-7B fine-tuned on MedMax · VQA tasks | accuracy: 23% | Helped |
| Add visual-chat instances to the training mixture (ablation) | Anole-7B fine-tuned on MedMax · Visual chat | LLM score: 17% | Helped |
| Replace the original VQGAN visual tokens with tokens from a biomedical-fine-tuned visual encoder (ablation) | Anole-7B fine-tuned on MedMax · VQA evaluation suite (average) | accuracy: 3% | Hurt |
| Fine-tune the VQGAN visual encoder on biomedical images (ablation) | VQGAN visual encoder · Held-out biomedical images | L1 reconstruction loss: 8.1 -> 7.8 | Helped |
Surprising findings
- Chameleon-7B achieves the highest QuiltVQA (Closed) score (71.4) but very low QuiltVQA (Open) accuracy, suggesting closed-ended questions are easier for the base model.
- Finetuning the VQGAN visual tokenizer on biomedical images and retokenizing data reduced downstream performance by about 3% relative to original visual tokens.
Limitations
- Limited to single-image settings; does not handle multi-image understanding or generation, and Chameleon's context length is restrictive (1024 tokens per image).
- Dataset scale limited by budget (about $500 in GPT-4o API spend for the interleaved generation data), and the model is a research prototype with occasional misinterpretations and artifacts in generated images.
- Open-ended VQA and multimodal generation rely on LLM-judge scoring (GPT-4o-mini), which may be subjective.
Experimental setup
Datasets: VQA-RAD SLAKE PathVQA PMC-VQA OmniMedVQA LLaVA-Med-PMC PMC-OA Quilt-1M PubMedVision LLaVA-Med-Instruct-120K Quilt-Instruct MIMIC-CXR
Benchmarks: VQA-RAD SLAKE PathVQA QuiltVQA PMC-VQA OmniMedVQA PathMMU ProbMed LLaVA-Med visual chat eval PMC-OA Quilt MIMIC-CXR
Metrics: accuracy (exact match) accuracy (LLM-judged) BioMedCLIPScore LLM score relative prediction score
Baselines: Chameleon-7B LLaVA-Med-v1.5 HuatuoGPT-Vision-7B GPT-4o GPT-4o-mini LLaVA-Med-v1
Builds on: Chameleon Anole LoRA LLaVA-Med PubMedVision
Closest papers by meaning
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.90)
- Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE (NeurIPS 2024 · similarity 0.89)
- Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering (AAAI 2025 · similarity 0.89)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.88)
- Biomedical Visual Instruction Tuning with Clinician Preference Alignment (NeurIPS 2024 · similarity 0.88)
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.