Research map / Detection and segmentation / Open-set and few-shot detection
SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates open-ended referring segmentation as set-level concept prediction: an LVLM emits hierarchical natural-language concepts that condition joint mask-set decoding, backed by a 236k-sample hierarchical annotation corpus.
Problem
Referring segmentation with multiple, cross-category, or open-ended targets; existing LVLM methods use one [SEG] token per target, treating targets as separate outputs and ignoring set-level properties like completeness and mutual exclusivity.
Approach
Replace implicit [SEG] tokens with LVLM-generated natural-language concepts (a set-level concept followed by fine-grained sub-category concepts) whose hidden states condition a SAM 3-based decoder for joint mask-set prediction, trained with a DETR-style set loss and a two-stage LLM-based hierarchical annotation pipeline.
Key results
- +3.3 gIoU on gRefCOCO val and +12.1 gIoU on MUSE val over prior best.
- +10.9 J&F on MeViS and +12.4 J&F on Ref-SeCVOS, new SOTA on all seven referring video benchmarks.
- Average 78.9 across RefCOCO/+/g and ReasonSeg, best overall average among compared methods.
Surprising findings
- Sa2VA-style [SEG] embeddings cluster more by spatial position than by semantic category, and multi-target performance degrades sharply with target count under the per-token paradigm.
Limitations
- Open-ended queries can have ambiguous target boundaries and concept granularity, causing failures.
- Model-assisted annotations may contain noise and have limited coverage of rare long-tail cases.
- Single-target benchmarks are only competitive (not best) on RefCOCO/+/g.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg MUSE MeViS Ref-DAVIS Ref-YouTube-VOS ReVOS
Benchmarks: gRefCOCO MUSE RefCOCO RefCOCO+ RefCOCOg ReasonSeg ReVOS ReasonVOS Ref-YTVOS Ref-DAVIS MeViS Ref-SeCVOS
Metrics: gIoU cIoU F1@0.5 J&F
Baselines: LISA GSVA PixelLM SAM4MLLM Text4Seg MLLMSeg SAM3 Agent VisionReasoner Sa2VA X-SAM VISA VideoLISA
Builds on: LISA SAM 3 Sa2VA DETR set prediction loss
Closest papers by meaning
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.88)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.88)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.87)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.87)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.87)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.