AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

RSAgent: Learning to Reason and Act via Multi-Turn Tool Invocations for Text-Guided Segmentation

Xingqi He, Yujie Zhang, Shuyong Gao, Wenjie Li, Lingyi Hong, Mingxi Chen, Kaixun Jiang, Jiyuan Fu, Wenqiang Zhang

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RSAgent, an agentic MLLM that performs multi-turn tool invocations to reason about and refine pixel-level segmentation prompts for text-guided segmentation.

Problem

Text-guided object segmentation needs cross-modal reasoning and pixel grounding, but most methods use a single forward pass that predicts pixel prompts for a segmentation model, leaving no room to verify, refocus, or refine when initial localization is wrong.

Approach

Recasts text-guided segmentation as an agentic process in which an MLLM interleaves reasoning with multi-turn calls to a segmentation tool, allowing verification and iterative refinement of localization rather than a single-pass prediction.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add RL after cold-start SFT (ablation)RSAgent · ReasonSeg testcIoU: 55.4 -> 57.9Improved
add RL after cold-start SFT (ablation)RSAgent · RefCOCOg testAcIoU: 73.9 -> 81.8Improved
add cold-start SFT before RL (ablation)RSAgent · ReasonSeg testcIoU: 54.3 -> 57.9Improved
add cold-start SFT before RL (ablation)RSAgent · RefCOCOg testAcIoU: 77.3 -> 81.8Improved
train the tool agent (SFT + RL) instead of off-the-shelf model (ablation)Qwen2.5-VL-7B tool agent · ReasonSeg testcIoU: 30.1 -> 57.9Improved
add final reward Rfinal (ablation)RSAgent · ReasonSeg testcIoU: 48.3 -> 57.9Improved
add process reward Rprocess (ablation)RSAgent · ReasonSeg testcIoU: 55.9 -> 57.9Improved
add format reward Rformat (ablation)RSAgent · ReasonSeg testcIoU: 54.7 -> 57.9Improved
use SAM2-hiera-large instead of SAM2-hiera-tiny as tool (ablation)RSAgent · ReasonSeg testcIoU: 53.1 -> 57.9Improved
use SAM2-hiera-large instead of SAM-H as tool (ablation)RSAgent · RefCOCOg testAcIoU: 74.6 -> 81.8Improved
use multi-turn tool invocation instead of single-pass output (ablation)RSAgent · ReasonSeg testgIoU: 56.1 -> 66.5Improved
increase max tool-call turns during training up to 8 (ablation)RSAgent · ReasonSeg testcIoUImproved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy