Research map / Vision-language and multimodal / Multimodal LLMs

DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DR²Seg, a self-rewarding GRPO framework for reasoning segmentation that decomposes each rollout into a think-then-description stage and a second pass on the description, using description-correctness and length self-rewards to curb overthinking without an external expert MLLM.

Problem

RL-based reasoning segmentation with MLLMs suffers from overthinking: verbose reasoning chains waste tokens and distract object localization. Prior fixes rely on an external large expert MLLM or thinking supervision.

Approach

A two-stage rollout where the MLLM first produces a self-contained referring description, then segments from that description alone; the second-pass answer accuracy gives a description self-reward and a comparative plus anchored length reward encourages shorter reasoning in the second pass, all without extra models or thinking supervision.

Key results

• On ReasonSeg val, DR²Seg* (Qwen2.5-VL-7B, few-shot) reaches 68.5 gIoU with 26.9 tokens versus VisionReasoner* 65.4 gIoU with 85.3 tokens.
• Zero-shot DR²Seg with only the description reward nearly halves reasoning tokens (81.5 to 56.1) and raises val gIoU from 64.9 to 67.6 in ablation.
• On the 3B model, DR²Seg reaches 65.5 gIoU vs 61.7 for VisionReasoner-3B with about half the tokens (23.9 vs 49.7).

Surprising findings

• Length reward is disabled in zero-shot training because training referring expressions are short, so Principle 2 cannot be reliably satisfied.
• Directly training VisionReasoner on the ReasonSeg train set degrades accuracy, while DR²Seg improves it.

Limitations

• Theoretical analysis (information-theoretic and variance arguments) is loose and relies on assumptions not verified empirically.
• Length anchor and penalty are sensitive: too small an anchor (N0=25) sharply increases tokens, and the length reward is inactive when all rollouts fail.
• Few-shot gains rely on small ReasonSeg training set of 239 samples; SAM3 reward uses an external API.

Experimental setup

Datasets: VisionReasoner-7K (LVIS, RefCOCOg, gRefCOCO, LISA++) ReasonSeg train

Benchmarks: ReasonSeg val ReasonSeg test RefCOCO RefCOCO+ RefCOCOg

Metrics: gIoU cIoU reasoning tokens

Baselines: VisionReasoner Seg-Zero SAM-R1 PixelThink LISA CoReS PixelLM Perception-GPT

Builds on: VisionReasoner GRPO Seg-Zero LISA

Closest papers by meaning

• Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.91)
• LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.91)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.89)
• SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation (CVPR 2026 · similarity 0.89)
• Discriminative Perception via Anchored Description for Reasoning Segmentation (CVPR 2026 · similarity 0.89)
• EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.88)