Research map / Vision-language and multimodal / Multimodal LLMs
RSAgent: Learning to Reason and Act via Multi-Turn Tool Invocations for Text-Guided Segmentation
Xingqi He, Yujie Zhang, Shuyong Gao, Wenjie Li, Lingyi Hong, Mingxi Chen, Kaixun Jiang, Jiyuan Fu, Wenqiang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RSAgent, an agentic MLLM that performs multi-turn tool invocations to reason about and refine pixel-level segmentation prompts for text-guided segmentation.
Problem
Text-guided object segmentation needs cross-modal reasoning and pixel grounding, but most methods use a single forward pass that predicts pixel prompts for a segmentation model, leaving no room to verify, refocus, or refine when initial localization is wrong.
Approach
Recasts text-guided segmentation as an agentic process in which an MLLM interleaves reasoning with multi-turn calls to a segmentation tool, allowing verification and iterative refinement of localization rather than a single-pass prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RL after cold-start SFT (ablation) | RSAgent · ReasonSeg test | cIoU: 55.4 -> 57.9 | Improved |
| add RL after cold-start SFT (ablation) | RSAgent · RefCOCOg testA | cIoU: 73.9 -> 81.8 | Improved |
| add cold-start SFT before RL (ablation) | RSAgent · ReasonSeg test | cIoU: 54.3 -> 57.9 | Improved |
| add cold-start SFT before RL (ablation) | RSAgent · RefCOCOg testA | cIoU: 77.3 -> 81.8 | Improved |
| train the tool agent (SFT + RL) instead of off-the-shelf model (ablation) | Qwen2.5-VL-7B tool agent · ReasonSeg test | cIoU: 30.1 -> 57.9 | Improved |
| add final reward Rfinal (ablation) | RSAgent · ReasonSeg test | cIoU: 48.3 -> 57.9 | Improved |
| add process reward Rprocess (ablation) | RSAgent · ReasonSeg test | cIoU: 55.9 -> 57.9 | Improved |
| add format reward Rformat (ablation) | RSAgent · ReasonSeg test | cIoU: 54.7 -> 57.9 | Improved |
| use SAM2-hiera-large instead of SAM2-hiera-tiny as tool (ablation) | RSAgent · ReasonSeg test | cIoU: 53.1 -> 57.9 | Improved |
| use SAM2-hiera-large instead of SAM-H as tool (ablation) | RSAgent · RefCOCOg testA | cIoU: 74.6 -> 81.8 | Improved |
| use multi-turn tool invocation instead of single-pass output (ablation) | RSAgent · ReasonSeg test | gIoU: 56.1 -> 66.5 | Improved |
| increase max tool-call turns during training up to 8 (ablation) | RSAgent · ReasonSeg test | cIoU | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SAM-R1 — SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- LENS: Learning to Segment Anything with Unified Reinforced Reasoning (AAAI 2026 · similarity 0.87)
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Socratic-Geo: Synthetic Data Generation and Cross-Modal Geometric Reasoning via Multi-Agent Interaction (CVPR 2026 · similarity 0.87)
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.87)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
