Research map / Vision-language and multimodal / Multimodal LLMs

Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Seg-ReSearch, an RL-trained MLLM agent that interleaves reasoning with external web search and a frozen mask generator to segment language-referred targets needing outside knowledge, plus a new OK-VOS benchmark.

Problem

Reasoning segmentation models are limited by the frozen internal knowledge of MLLMs and cannot handle queries needing up-to-date or domain-specific information.

Approach

Train an MLLM to alternate visual reasoning and external search (text/image) via GRPO, using a hierarchical reward combining an initial-guidance reward on the first search query, a tapering format-based process reward that discourages infinite loops, and binary/continuous outcome rewards for keyframe selection and localization.

Key results

• On OK-VOS, Seg-ReSearch-8B reaches 50.0 overall J&F versus 38.2 for Qwen3-VL-8B*+Search and 29.6 for GLUS-7B.
• Seg-ReSearch-8B achieves 63.2 J&F on ReasonVOS, +8.3 over OneThinker-8B (54.9).
• On ReasonSeg test, Seg-ReSearch-8B reaches 67.4 gIoU and 59.0 cIoU, above SAM3-Agent-7B (62.6 / 56.2).

Surprising findings

• Simply adding search to Qwen3-VL-8B improves OK-VOS J&F by only about 1.8 points.
• A linear per-search process bonus causes reward hacking, dropping J&F to 41.1 as the policy abuses search.

Limitations

• OK-VOS is small (150 videos, 1,000 queries) and training uses only 100 manually annotated samples.
• Search results depend on a live search API (Google), limiting reproducibility over time.
• Mask quality depends on a frozen external mask generator (SAM2 or SeC).

Experimental setup

Datasets: OK-VOS ReasonSeg ReasonVOS 100 manually annotated training samples

Benchmarks: OK-VOS ReasonSeg ReasonVOS

Metrics: J&F J F gIoU cIoU

Baselines: Qwen3-VL-8B* Qwen3-VL-8B*+Search Qwen3-VL-4B*+Search SAMWISE ReferDINO ReferEverything-1.4B VideoLISA-3.8B GLUS-7B RGA3-7B UniPixel-7B OneThinker-8B LISA-7B

Builds on: GRPO Search-R1 LISA Seg-Zero SAM2

Closest papers by meaning

• DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.91)
• OMG-Seg: Is One Model Good Enough For All Segmentation? (CVPR 2024 · similarity 0.90)
• SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation (CVPR 2026 · similarity 0.89)
• One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · similarity 0.89)
• Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
• LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.87)