Research map / Vision-language and multimodal / Multimodal LLMs

MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MedSIGHT, a medical LVLM on Qwen3-8B that adds a Region Perceiver, a modality-aware region codebook in the LLM vocabulary, and progressive training to unify medical VQA and pixel-level segmentation, trained on 72K instruction pairs.

Problem

Medical LVLMs handle either visual comprehension or pixel-level grounding/segmentation but struggle to unify them; existing unified models use patch-level CLIP features and a single [SEG] token for all regions.

Approach

A Region Perceiver with dual cross-attention produces region-centric tokens with spatial detail; a modality-aware vector-quantized region codebook is added to the LLM vocabulary so the LLM emits discrete region codes that are decoded to masks via a text-to-vision projector, trained through a progressive multi-stage pipeline.

Key results

• On medical VQA comprehension, MedSIGHT averages 62.3 across comprehension benchmarks, above the best baseline HuatuoGPT-Vision (58.3), with 72K instruction samples versus 647K.
• On DiagSeg, MedSIGHT reaches mean Dice 69.9 for segmentation and 58.9 recall for diagnosis, versus MedPLIB at 31.8 and 13.1 respectively.
• On MeCoVQA-G text-prompted segmentation, MedSIGHT achieves 42.8 mean Dice, above MedPLIB (40.1).

Surprising findings

• MedSIGHT underperforms MedPLIB on the ultrasound modality of DiagSeg-Diagnosis (51.7 vs 71.9), which the authors attribute to training-data overlap with SA-Med2D-20M.
• Removing the grounding instruction tuning stage causes the model to fail to output codebook tokens for segmentation.

Limitations

• Codebook may generalize poorly to unseen modalities and regions, as acknowledged in the conclusion.
• Results on several out-of-distribution MeCoVQA-G modalities (e.g., PET, US) are lower than baselines.
• DiagSeg relies on GPT-5-generated QA pairs from test-set masks and descriptions; validation covers only 100-240 sampled cases.

Experimental setup

Datasets: BiomedParse PubMedVision MeCoVQA-Grounding SLAKE VQA-RAD PathVQA OmniMedVQA MMMU-Med

Benchmarks: SLAKE VQA-RAD PathVQA OmniMedVQA MMMU-Med DiagSeg MeCoVQA-G S-Chain MedTrinity-25M

Metrics: accuracy recall Dice mIoU

Baselines: LISA LISA++ LaSagnA GLaMM OMG-LLaVA MedPLIB HuatuoGPT-Vision Lingshu-7B HealthGPT LLaVA-Med InternVL3.5 Qwen3-VL

Builds on: LISA MedPLIB Mask2Former VQ-VAE BiomedParse

Closest papers by meaning

• Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.90)
• HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.90)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.90)
• MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.90)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.90)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.89)