Research map / Vision-language and multimodal / Multimodal LLMs
SAM-Veteran: An MLLM-Based Human-like SAM Agent for Reasoning Segmentation
Tianyuan Du, Haopeng Li, Zhen Fan, Jiarui Zhang, Panwang Pan, Yang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-Veteran, an MLLM-based agent that uses SAM as an interactive tool for iterative mask refinement in reasoning segmentation.
Problem
Existing MLLM+SAM reasoning segmentation paradigms do not exploit SAM's ability to support iterative, interactive mask refinement.
Approach
An MLLM agent that behaves more like a human annotator, interacting with SAM iteratively to refine masks rather than producing a single prompt-to-mask pass.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mask comprehension task to RL training (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 70.3 -> 72.1 | Improved |
| add auxiliary corrupted-mask task (ablation) | SAM-Veteran · 5-benchmark average | avg IoU; termination: 72.1 -> 72.2; Never -> Adaptive termination | Improved |
| add textual grounding RL training to Qwen2.5-VL (ablation) | Qwen+SAM2 · 5-benchmark average | avg IoU: 64.4 -> 70.3 | Improved |
| add SAM reward (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 71.3 -> 72.2 | Improved |
| add decision reward (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 70.6 -> 72.2 | Improved |
| add IoU improvement reward (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 70.4 -> 72.2 | Improved |
| use graded IoU improvement reward instead of hard version (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 71.9 -> 72.2 | Improved |
| add dynamic sampling (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 70.8 -> 72.2 | Improved |
| add chain of thought in all tasks (ablation) | SAM-Veteran · ReasonSeg val | gIoU: 64.4 -> 68.2 | Improved |
| add chain of thought in mask tasks (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 71.6 -> 72.2 | Improved |
| iterative refinement from the trained model vs Qwen | SAM-Veteran · ReasonSeg val | gIoU change over steps: 0.00 -> 2.47 (Qwen: 0.00 -> -2.48) | Improved |
| use hard IoU box reward threshold 0.5 instead of soft IoU reward (ablation) | SAM-Veteran · 5-benchmark average | avg IoU: 70.7 -> 72.3 | Improved |
Later papers that compared with this method
1 later accepted paper reports SAM-Veteran as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation (ECCV 2026 · better on 2 of 2 shared results)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SAM-R1 — SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Builds on: Segment Anything Model
Closest papers by meaning
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.89)
- SAMTok: Representing Any Mask with Two Words (CVPR 2026 · similarity 0.89)
- LENS: Learning to Segment Anything with Unified Reinforced Reasoning (AAAI 2026 · similarity 0.87)
- CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation (ICLR 2026 · similarity 0.87)
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
