Research map / Vision-language and multimodal / Multimodal LLMs

MedSG-Bench: A Benchmark for Medical Image Sequences Grounding

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedSG-Bench, a 9,630-pair VQA-style benchmark for grounding across medical image sequences (eight tasks in image-difference and image-consistency paradigms), plus MedSG-188K training data and the MedSeq-Grounder model fine-tuned from Qwen2.5-VL-7B.

Problem

Existing medical visual grounding benchmarks focus on single images, leaving grounding across sequential medical images (pre/post treatment, multi-view, multi-modal, slices/frames) underexplored.

Approach

A benchmark of eight VQA-style grounding tasks over medical image sequences, grouped into image difference grounding and image consistency grounding, with a grounding instruction-tuning dataset (MedSG-188K) and an MLLM (MedSeq-Grounder) trained on it.

Key results

• Best baseline Migician (7B) reaches average IoU 20.29% and Acc@0.5 11.39% on MedSG-Bench.
• MedSeq-Grounder (Qwen2.5-VL-7B fine-tuned on MedSG-188K) reaches average IoU 72.55% and Acc@0.5 79.71%.
• HuatuoGPT-Vision-7B averages 8.97% IoU, trailing Qwen2.5-VL-7B (12.31%) and even Qwen2.5-VL-3B (10.94%).

Measured effects reported in the paper

Surprising findings

• Medical-specialized HuatuoGPT-Vision underperforms general-purpose Qwen2.5-VL, which the authors attribute to catastrophic forgetting of grounding ability.
• Larger or newer models do not reliably improve grounding; InternVL2-8B beats InternVL2.5-8B and InternVL3-8B, and the 76B InternVL2 underperforms its 8B counterpart.

Limitations

• Evaluation is zero-shot only, with no in-context examples tested.
• Many tasks rely on synthetic perturbations (CutPaste, generative anomalies), which may not reflect real clinical change.
• Tasks like Visual Patch Grounding use manual selection of patches, limiting scalability.

Experimental setup

Datasets: MedSG-188K 76 public medical datasets (e.g., BraTS2020, VinDr-Mammo, ACDC, EndoVis2017)

Benchmarks: MedSG-Bench

Metrics: IoU Acc@0.5

Baselines: Qwen2.5-VL MiniCPM-V-2_6 MiniCPM-O-2_6 mPLUG-Owl3 Mantis-Idefics2 LLaVA-OneVision InternVL2 InternVL2.5 InternVL3 Migician HuatuoGPT-Vision

Builds on: Qwen2.5-VL Migician LLaMA-Factory CutPaste

Closest papers by meaning

• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.93)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.90)
• MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.90)
• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.88)
• Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.87)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.87)