Research map / Detection and segmentation / Open-set and few-shot detection
Exploring Simple Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-Seg, a MaskFormer trained from DINO-clustered pseudo-masks and image-text contrastive loss on public web pairs, achieving competitive open-vocabulary segmentation without CLIP, SAM, or GT masks.
Problem
Open-vocabulary semantic segmentation requires pixel-level image-text alignment, but existing methods rely on CLIP, ground-truth masks, custom grouping encoders, or SAM.
Approach
Decouple mask and semantic supervision: train a MaskFormer with class-agnostic pseudo-masks from K-means clustering of DINO ViT-S/8 tokens for mask supervision, and an image-text contrastive loss on averaged mask features for language alignment, with no CLIP or annotated masks; optional self-training yields S-Seg+.
Key results
- S-Seg+ reaches 62.0 mIoU on Pascal VOC with background, versus 53.2 for S-Seg and 55.0 for TCL under the same protocol.
- Self-training (S-Seg+) improves the 3-dataset average by 5.5% mIoU over S-Seg (42.6 vs 37.1).
- Without background, S-Seg achieves 81.8 on Pascal VOC and 42.4 on COCO, outperforming GroupViT (77.2 and 37.5) on these datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-training to S-Seg (ablation) | S-Seg · PASCAL VOC (background included) | mIoU: 53.2 -> 62.0 | Helped |
| Add self-training to S-Seg (ablation) | S-Seg · PASCAL Context (background included) | mIoU: 27.9 -> 30.2 | Helped |
| Add self-training to S-Seg (ablation) | S-Seg · COCO (background included) | mIoU: 30.3 -> 35.7 | Helped |
| Add self-training to S-Seg (ablation) | S-Seg · PASCAL VOC (background excluded) | mIoU: 81.8 -> 84.7 | Helped |
| Add self-training to S-Seg (ablation) | S-Seg · PASCAL Context (background excluded) | mIoU: 27.2 -> 31.6 | Helped |
| Add self-training to S-Seg (ablation) | S-Seg · COCO (background excluded) | mIoU: 42.4 -> 53.0 | Helped |
| Increase image-text training data from 12M to 26M (ablation) | S-Seg and S-Seg+ · PASCAL VOC, PASCAL Context, and COCO | mIoU | Helped |
Surprising findings
- Simple baselines (pseudo-masks + CLIP classification, or pseudo-mask ViT) perform poorly, showing that classifying high-quality masks alone is not sufficient.
Limitations
- Performance on Pascal Context and COCO with background trails fully supervised and some CLIP-based methods
- Simple baselines suggest decomposed segment-then-classify pipelines fail, so the approach relies on joint training
Experimental setup
Datasets: CC3M CC12M RedCaps
Benchmarks: Pascal VOC 2012 Pascal Context COCO
Metrics: mIoU
Baselines: GroupViT MaskCLIP CLIP TCL OVSegmentor SegCLIP SAM-CLIP ZegFormer
Builds on: MaskFormer CLIP ZegFormer GroupViT DINO
Closest papers by meaning
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.92)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.90)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.90)
- OMG-Seg: Is One Model Good Enough For All Segmentation? (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.