Research map / Vision-language and multimodal / Multimodal LLMs
LENS: Learning to Segment Anything with Unified Reinforced Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LENS, an end-to-end GRPO-based RL framework that couples Qwen2.5-VL-3B with SAM2 via a context-query connector, using unified sentence/box/segment rewards to get chain-of-thought-guided text-prompted segmentation.
Problem
Existing MLLM-based text-prompted (referring/reasoning) segmentation methods use SFT with a single seg token, ignoring test-time chain-of-thought reasoning and generalizing poorly to unseen prompts and domains.
Approach
Couple an MLLM with SAM2 through learnable context queries and a shallow transformer connector, train with a GRPO objective using format, box-IoU and segment-IoU rewards plus a segmentation loss, so CoT reasoning and mask quality are optimized jointly end-to-end.
Key results
- LENS-3B reaches 81.2% average cIoU across RefCOCO, RefCOCO+, and RefCOCOg, versus 75.6% for GLaMM.
- LENS-3B achieves 58.0% cIoU on ReasonSeg-Test and 78.3% cIoU on GroundingSuite-Eval, with the latter 9.4 points above Seg-Zero-3B.
- Ablation on RefCOCO testA: adding the alignment stage raises cIoU from 81.9% to 82.7% after segment reward, with a base of 79.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RL (GRPO with unified rewards) after alignment (adds R_seg) (ablation) | context module + alignment pipeline · RefCOCO testA | cIoU: +1.8 (80.9 to 82.7) | Helped |
| Add pretraining alignment stage to context module (ablation) | context module + box/format rewards · RefCOCO testA | cIoU: +9.6 (72.3 to 81.9) | Helped |
| Add context module (no alignment, no RL) (ablation) | Lisa-like baseline · RefCOCO testA | cIoU: +1.1 (79.3 to 80.4) | Helped |
| Add box and format rewards without alignment stage (directly adding context module with RL) (ablation) | context module RL training · RefCOCO testA | cIoU: 72.3 vs 80.9 with alignment | Hurt |
| Replace 4-layer transformer connector with MLP connector (ablation) | pretraining alignment connector · RefCOCO testA | cIoU: -15.0 (87.8 to 72.8) | Hurt |
| Increase context queries from 1 to 64 (ablation) | context module · RefCOCO testA (alignment stage) | cIoU: +2.0 (85.9 to 87.9) | Helped |
| Increase context queries from 64 to 128 (ablation) | context module · RefCOCO testA (alignment stage) | cIoU: -0.7 (87.9 to 87.2) | Hurt |
| Switch SAM1 frozen to SAM2 with tuned mask decoder (no context module) (ablation) | Lisa-like baseline · RefCOCO testA | cIoU: +0.8 (78.5 to 79.3) for SAM2 tuned vs SAM1 frozen; +0.6 for tuning SAM1 (78.5 to 79.1) | Helped |
| Use LENS end-to-end (context module + RL) vs Seg-Zero's frozen SAM with boxes/points | LENS-3B vs Seg-Zero-3B · GroundingSuite-Eval | cIoU: 78.3 vs 68.9 (+9.4) | Helped |
| Use LENS-3B vs Seg-Zero-3B on ReasonSeg-Test | LENS-3B · ReasonSeg-Test | cIoU: 58.0 vs 48.6 | Helped |
| Use LENS-3B (RL + context) vs GLaMM on RefCOCO series average | LENS-3B · RefCOCO/RefCOCO+/RefCOCOg average cIoU | cIoU: 81.2 vs 75.6 | Helped |
Surprising findings
- Adding the box reward together with the format reward without alignment yields unstable training and a drop to 72.3% cIoU.
- Replacing an MLP connector with a 4-layer transformer connector raises cIoU from 72.8% to 87.8%, a larger effect than most other components.
Limitations
- Fails on fine-grained part-level granularity (e.g., segments both plate and fruit), wrong-instance selection, and very small objects.
- Reported gains rely on a RefCOCO-based training split, and several baselines (PSALM, HyperSeg, InstructSeg) are trained on COCO-overlapping data, complicating comparison.
- Ablation and comparisons rely on single-run results with no variance reported.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReasonSeg COCO 2017
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg GroundingSuite-Eval
Metrics: cIoU gIoU
Baselines: LISA PixelLM GLaMM OMG-LLaVA Seg-Zero-3B Seg-Zero-7B PSALM HyperSeg InstructSeg SAM3-Agent
Builds on: GRPO VLM-R1 LISA Seg-Zero SAM2
Closest papers by meaning
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.88)
- AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.88)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.88)
- Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.