Research map / Vision-language and multimodal / Multimodal LLMs
SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-R1, an RL framework (modified GRPO) that trains Qwen2.5-VL-7B for reasoning segmentation using SAM-derived IoU rewards, reaching strong results with only 3k RefCOCOg training samples.
Problem
Reasoning segmentation with multimodal LLMs usually requires costly reasoning-annotated data and SFT, which risks catastrophic forgetting and poor out-of-domain generalization; decoupled reward designs also lack pixel-level feedback.
Approach
Places SAM directly in the RL loop as a reward provider, computing tiered IoU rewards from SAM masks for the MLLM's predicted box/point outputs, combined with format rewards and a modified GRPO objective (asymmetric clipping with higher upper bound and token-level loss normalization).
Key results
- On ReasonSeg zero-shot test, SAM-R1 reaches 60.2 gIoU and 54.3 cIoU vs. Seg-Zero-7B 58.3 gIoU and 53.4 cIoU, using only 3k training samples.
- On RefCOCO+ testA cIoU, SAM-R1 reaches 74.7 vs. Seg-Zero-7B 73.9; on RefCOCOg it reaches 73.1 vs. 73.3 for Seg-Zero.
- On LISA-Grounding, SAM-R1 scores 63.8 vs. Visual-RFT 43.9 and GroundedSAM 26.2 without REC training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAM-R1 rather than Seg-Zero-7B* | Qwen2.5VL-7B reasoning segmentation · ReasonSeg-zero-shot test | gIoU: 58.3 -> 60.2 | Helped |
| Use SAM-R1 rather than Seg-Zero-7B* | Qwen2.5VL-7B reasoning segmentation · ReasonSeg-zero-shot test | cIoU: 53.4 -> 54.3 | Helped |
| Use SAM-R1 rather than Seg-Zero-7B* | Qwen2.5VL-7B referring expression segmentation · RefCOCO+ testA | cIoU: 73.9 to 74.7 | Helped |
| Use SAM-R1 rather than Seg-Zero-7B* | Qwen2.5VL-7B referring expression segmentation · RefCOCOg test | cIoU: 73.3 -> 73.1 | Hurt |
| Use SAM-R1 rather than Seg-Zero-7B* | Qwen2.5VL-7B referring expression segmentation · RefCOCO testA | cIoU: 79.2 -> 79.2 | No clear effect |
| Replace fixed thresholds with tiered segmentation-accuracy thresholds (ablation) | SAM-R1 · ReasonSeg-zero-shot test | gIoU: 58.6 -> 60.2 | Helped |
| Add token-level loss normalization to GRPO (ablation) | SAM-R1 training · ReasonSeg-zero-shot test | cIoU: 51.2 -> 51.7 | Helped |
| Increase the upper clipping threshold in GRPO (ablation) | SAM-R1 training · ReasonSeg-zero-shot test | gIoU: 57.8 -> 59.1 | Helped |
| Add token-level loss normalization and higher clipping threshold to GRPO (ablation) | SAM-R1 training · ReasonSeg-zero-shot test | cIoU: 51.2 -> 54.3 | Helped |
| Increase training samples from 3k to 10k (ablation) | SAM-R1 · ReasonSeg-zero-shot test | cIoU: 54.3 to 53.9 | Hurt |
| Remove the KL divergence constraint (ablation) | SAM-R1 reinforcement learning · Training | training stability | Hurt |
| Reward outputs for including both positive and negative reference points (ablation) | SAM-R1 reward design · Reference-point generation | negative-point placement | Hurt |
Surprising findings
- Removing the KL constraint caused training collapse after about 100 steps.
- Rewarding negative reference points led the model to place negatives outside the target box, so the model was not encouraged to generate useful negative points.
Limitations
- SAM is frozen, so information flows one way and SAM cannot adapt to the reasoning model.
- Model struggles to generate meaningful negative reference points.
- Failure cases include incomplete segmentation and over-segmentation of parts.
Experimental setup
Datasets: RefCOCOg RefCOCO RefCOCO+ ReasonSeg LISA-Grounding
Benchmarks: ReasonSeg RefCOCO testA RefCOCO+ testA RefCOCOg test LISA-Grounding
Metrics: gIoU cIoU
Baselines: Seg-Zero-7B LISA-7B LISA-13B PixelLM-7B PerceptionGPT-7B SAM4MLLM Grounded-SAM Visual-RFT
Builds on: GRPO DeepSeek-R1 Seg-Zero VLM-R1 LISA
Closest papers by meaning
- LENS: Learning to Segment Anything with Unified Reinforced Reasoning (AAAI 2026 · similarity 0.91)
- OneThinker: All-in-one Reasoning Model for Image and Video (CVPR 2026 · similarity 0.89)
- SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning (NeurIPS 2025 · similarity 0.89)
- FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.