Research map / Vision-language and multimodal / Multimodal LLMs

IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IBISAgent, a Qwen2.5-VL-7B agent that performs multi-turn click-based segmentation with MedSAM2 as a tool, trained by cold-start SFT then GRPO with fine-grained rewards for biomedical referring and reasoning segmentation.

Problem

Medical MLLMs for pixel-level segmentation rely on implicit <SEG> tokens and external decoders, causing catastrophic forgetting and poor generalization, and they use single-pass reasoning without iterative mask refinement.

Approach

Reformulate biomedical segmentation as a multi-step MDP where the MLLM emits interleaved reasoning and text-based click actions, invokes an interactive segmenter (MedSAM2) as a tool, and refines masks over turns; trained with cold-start SFT on click-simulated trajectories and GRPO with region-based click, progressive improvement, and length rewards.

Key results

• On the in-domain test set IBISAgent reaches 85.58 IoU and 92.21 DSC, versus 50.74 IoU for the best medical baseline (UniBiomed).
• On MeCOVQA-G+ IBISAgent reaches 80.63 IoU and 95.24 F1, versus 46.54 IoU for Citrus-V.
• Against GPT-4o+MedSAM2 on the in-house set, IBISAgent gets 72.09 IoU versus 7.23 IoU; RL alone (M_rl) gives 62.77 IoU vs 53.42 for cold-start SFT alone on the in-house set.

Surprising findings

• Ablation shows RL alone (no SFT) outperforms cold-start SFT alone on the in-house set (62.77 vs 53.42 IoU).

Limitations

• Multi-round refinement increases inference time (28.70 s per case) relative to single-pass MLLMs.
• Relies on MedSAM2 as segmentation tool and on proprietary models (Gemini-2.5-Pro, GPT-5) for data generation.

Experimental setup

Datasets: BiomedParseData Medical Segmentation Decathlon PathVQA SLAKE VQA-RAD OmniMedVQA PMC_VQA Huatuo_PubMedVision MeCOVQA-G+

Benchmarks: BiomedParseData in-domain test (D_test) MeCOVQA-G+ in-house held-out set

Metrics: IoU DSC F1-score inference time

Baselines: LISA LISA++ SAM4MLLM VisionReasoner MedPLIB Citrus-V UniBiomed MMedAgent GPT-4o + MedSAM2

Builds on: LISA GRPO MedSAM2 VERL Qwen2.5-VL

Closest papers by meaning

• SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories (CVPR 2025 · similarity 0.88)
• UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.87)
• PixelLM: Pixel Reasoning with Large Multimodal Model (CVPR 2024 · similarity 0.87)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.87)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.87)
• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.87)