Research map / Vision-language and multimodal / Visual question answering

MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output

CVPR 2025 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MIMO, a medical VLM that accepts visual prompts (points/boxes) as input and outputs text interleaved with pixel-grounded segmentation masks, trained on the new 895K-sample MIMOSeg dataset.

Problem

Medical vision-language models only take text queries over the whole image and give text-only answers, so they cannot refer to image regions in inputs or ground medical terms to masks in outputs.

Approach

Combine a visual prompt encoder and a multi-modal input aligner (cross-attention with learnable queries) on the input side with LLM [SEG] tokens decoded by a SAM mask decoder on the output side, so the model both answers using visual referring inputs and grounds terms in its answer.

Key results

• On the held-in MIMOSeg Perspective I, MIMO reaches mIoU 0.639 vs 0.556 for GLaMM in language-guided segmentation.
• On held-out VQA, MIMO averages 56.1% accuracy across VQA-RAD, SLAKE and PathVQA, above LLaVA-Med (52.3) and Qwen-VL-Chat (52.7) but below HuatuoGPT-Vision (66.1), which used more training data.
• On held-out X-ray segmentation without bounding boxes, MIMO reaches AP50 0.507 vs 0.335 for GLaMM.

Measured effects reported in the paper

Surprising findings

• MIMO without the multi-modal input aligner scores higher than full MIMO on several held-in perspectives (e.g., Perspective III ROUGE-L 0.650 vs 0.581), so the aligner does not uniformly help text metrics.

Limitations

• Q&A generation for MIMOSeg is bounded by the coverage of the manually built knowledge base, limiting information coverage.
• Held-out VQA comparison is limited to closed-ended questions, and HuatuoGPT-Vision was trained on more data, so the comparison is not like-for-like.

Experimental setup

Datasets: MIMOSeg LLaVA-Med VQA Slake RadVQA PathVQA X-ray Fundus Skin Lesion

Benchmarks: MIMOSeg test set Slake VQA-RAD PathVQA Fundus X-ray Skin Lesion

Metrics: BLEU-4 ROUGE-L METEOR AP50 mIoU F1 accuracy

Baselines: SAM-b SAM-h SAM-Med GLaMM LLaVA-Med HuatuoGPT-Vision GPT-4o Med-Flamingo RAD-FM Qwen-VL-Chat

Builds on: LLaVA-Med SAM GLaMM Vicuna

Closest papers by meaning

• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.92)
• MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.90)
• Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.90)
• GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.90)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.90)
• Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.90)