Research map / Vision-language and multimodal / Multimodal LLMs

Ophiuchus: Incentivizing Tool-augmented ''Think with Images'' for Joint Medical Segmentation, Understanding and Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Ophiuchus trains a Qwen2.5-VL-7B medical MLLM to think with images by interleaving calls to zoom-in, SAM2 and BiomedParse tools, using cold-start SFT, self-reflection fine-tuning and agentic tool RL with GRPO.

Problem

Medical MLLMs reason only in text and cannot dynamically inspect fine-grained image regions, which hurts VQA, detection and reasoning-based segmentation.

Approach

Equip an MLLM with segmentation and zoom-in tools inside its chain of thought and train it in three stages (cold-start SFT, self-reflection SFT on trajectories where the tool sequence changed and correctness flipped, and RL with format, answer, and conditional tool-use rewards).

Key results

• Ophiuchus-7B reaches 68.0 average over seven out-of-domain VQA benchmarks, versus 49.3 for Qwen2.5-VL-7B and 59.0 for GPT-5 (zero-shot).
• On In-House-Seg, Ophiuchus achieves 67.5 DSC versus 65.8 for SAM 2 standalone and 66.2 for the SAM 2 + BiomedParse ensemble.
• Ablation: RL-only reaches 62.8 average and cold-start + RL reaches 65.7; full three-stage training reaches 68.1.

Surprising findings

• Ophiuchus invokes necessary tools on 93.7% of cases the base model fails, and makes unnecessary tool calls only 3.3% of the time.
• Zero-shot swap of SAM2 for MedSAM, SAM-Med2D or SAM yields only minor performance changes (67.3 to 68.1 average).

Limitations

• Training data depends on proprietary teacher models (Gemini-2.5-Pro, GPT-5) for QA and trajectory generation.
• Tool set limited to zoom-in, SAM2 and BiomedParse; authors acknowledge tool quality bounds extensibility claims.
• Segmentation evaluation may be affected by data leakage concerns from SAM2 pretraining data.

Experimental setup

Datasets: BiomedParseData Malenia PathVQA SLAKE VQA-RAD OmniMedVQA MMMU (H&M) MedXpertQA

Benchmarks: D_test-VQA PathVQA SLAKE VQA-RAD OmniMedVQA MMMU (H&M) MedXpertQA In-House-VQA In-Domain-Seg In-House-Seg

Metrics: accuracy DSC

Baselines: GPT-4.1 GPT-5 OpenAI-o3 Gemini 2.5 Pro InternVL3-8B Qwen2.5-VL-32B DeepEyes-7B Mini-o3-7B-v1 PixelReasoner-RL-v1-7B LLaVA-Med-7B HuatuoGPT-Vision-34B Med-R1-2B

Builds on: GRPO Qwen2.5-VL SAM2 BiomedParse DeepEyes VERL

Closest papers by meaning

• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.87)
• IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (CVPR 2026 · similarity 0.86)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.86)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.86)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.86)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.86)