Research map / Detection and segmentation / Open-set and few-shot detection

Scaling up Image Segmentation across Data and Tasks

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary semantic segmentation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MQ-Former, a scalable segmentation transformer whose mixed query fuses learnable and conditional queries, trained across many datasets and tasks with synthetic SAM masks and captions to improve open-vocabulary generalization.

Problem

Existing segmentation models do not scale jointly across datasets with different class structures and across tasks (instance, semantic, panoptic, referring, foreground), and generalize poorly to open-vocabulary, free-form, in-the-wild segmentation; human mask annotation is also costly.

Approach

A mixed query mechanism that fuses learnable queries (good for stuff) and conditional queries (good for things) via conditional-to-learnable and learnable-to-conditional cross-attention, with a unified (class text, box, mask) training format and offline synthetic masks (SAM) and captions (OFA-like) for data expansion.

Key results

• Mixed query improves SeginW Mask AP from 27.8 (conditional) to 32.1-38.4 across scaling settings, and reaches 38.6 with dataset/task scaling; scaling to 2.2M images/6 tasks yields 43.2 SeginW, a 4.6-point gain from synthetic data.
• On closed-set COCO panoptic PQ 56.8 and ADE panoptic/semantic results, mixed query beats learnable and conditional variants by 1.3-2.7 PQ on COCO.
• MQ-Former achieves 58.8 PQ on COCO, 52.3 instance Mask AP, and 65.0 mIoU on PC-59 (open-set), outperforming X-Decoder and OpenSeeD among generalist models.

Measured effects reported in the paper

Surprising findings

• Conditional queries alone degrade ADE semantic segmentation by over 7 mIoU points, while learnable queries alone are weak on instance segmentation.
• Query selection is not strictly thing/stuff: conditional queries capture some stuff regions (e.g., sky, floor) and learnable queries capture some thing regions (e.g., table, car).

Limitations

• Synthetic captions and masks are generated offline by models (SAM, OFA-akin) and their quality limits the training signal.
• Instance segmentation performance on some settings remains below specialist models; comparable rather than superior to specialist SOTA on closed-set benchmarks.

Experimental setup

Datasets: COCO ADE20K LVIS Visual Genome Objects365 RefCOCO RefCOCO+ RefCOCOg HRSOD Pascal Context BDD SeginW

Benchmarks: SeginW COCO ADE20K Pascal Context BDD RefCOCOg RefCOCO UHRSD LVIS

Metrics: Mask AP PQ mIoU Box AP

Baselines: X-Decoder OpenSeeD DaTaSeg OneFormer Mask2Former Mask DINO ODISE OMG-Seg UNINEXT

Builds on: Mask DINO X-Decoder OpenSeeD Conditional DETR SAM

Closest papers by meaning

• Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
• LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.85)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.85)
• USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.85)
• Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.85)
• DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data (CVPR 2024 · similarity 0.85)