Research map / Vision-language and multimodal / Visual question answering
Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MedPLIB, an end-to-end biomedical MLLM with pixel-level understanding (VQA, visual prompts, pixel grounding) trained via a multi-stage MoE strategy, and introduces the MeCoVQA dataset.
Problem
Current biomedical MLLMs handle only image-level understanding and text-only interaction, lacking pixel-level prompting and grounding, and pixel-level medical VQA data is scarce.
Approach
Two experts (vision-language and pixel-grounding) are trained separately in multi-stage fashion, then combined as a top-1 routed MoE in the FFN and fine-tuned with LoRA, with a LISA-style SEG token driving a SAM-Med2D mask decoder and region tokens for visual prompts.
Key results
- MedPLIB achieves 60.58 mean accuracy on OmniMedVQA, leading the second-best BLIP-2 (54.62) by about 6 points.
- On MeCoVQA-G, MedPLIB reaches 68.11 mDice vs. LISA's 62.74, a 5.37-point gain.
- In zero-shot grounding across five unseen modalities, MedPLIB gets 32.27 mean mDice vs. LISA's 16.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard FFN with MoE (ablation) | MedPLIB · Ablation evaluation (four datasets) | Mean: 51.62 -> 54.24 | Helped |
| add Stage I alignment to Stage IV training (ablation) | MedPLIB · Ablation evaluation (four datasets) | Mean: 54.24 -> 54.42 | Helped |
| initialize Stage IV with the Stage II vision-language expert (ablation) | MedPLIB · Ablation evaluation (four datasets) | Mean: 54.42 -> 54.53 | Helped |
| initialize Stage IV with the Stage III grounding expert (ablation) | MedPLIB · Ablation evaluation (four datasets) | Mean: 54.42 -> 54.91 | Helped |
| add both Stage II recognition and Stage III grounding expert initialization (ablation) | MedPLIB · Ablation evaluation (four datasets) | Mean: 54.24 -> 56.83 | Helped |
| use top-2 dense routing instead of top-1 sparse routing (ablation) | MedPLIB · Ablation evaluation | performance | Hurt |
| set Capacity Factor to 1.5 rather than 1 or 2 (ablation) | MedPLIB · Ablation evaluation | performance | Helped |
| add pixel grounding capability alongside VQA | MedPLIB · MeCoVQA-C Test | recall: 52.83 -> 49.41 | Hurt |
Surprising findings
- Small specialist segmentation models (e.g., DMMI, LViT) remain stronger than MedPLIB on in-domain MeCoVQA-G grounding.
- MedPLIB slightly underperforms LISA in recall on complex VQA, attributed to balancing grounding capability.
Limitations
- Hallucinations in generated answers.
- Substantial errors in pixel grounding of small targets.
- Lags specialist segmentation models on in-domain grounding.
Experimental setup
Datasets: LLaVA-Med-alignment SLAKE PathVQA PMC-VQA ImageClef2021 ImageClef2019 VQA-RAD SA-Med2D-20M OmniMedVQA
Benchmarks: OmniMedVQA MeCoVQA-C MeCoVQA-R MeCoVQA-G
Metrics: accuracy precision recall mDice
Baselines: MiniGPT-4 BLIP-2 InstructBLIP LLaVA VPGTrans RadFM LLaVA-Med LISA LViT ReclMIS LAVT DMMI
Builds on: LISA LLaVA LLaVA-Med SAM-Med2D MoE-LLaVA
Closest papers by meaning
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.92)
- Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.91)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.91)
- SMMILE: An expert-driven benchmark for multimodal medical in-context learning (NeurIPS 2025 · similarity 0.91)
- GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.90)
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.