Research map / Detection and segmentation / Open-set and few-shot detection
Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes the CLIP image encoder for open-vocabulary semantic segmentation using only unlabeled SAM/DINO masks on SA-1B images, with globally clustered learnable class prompts (online Sinkhorn clustering) replacing semantic labels.
Problem
Open-vocabulary semantic segmentation normally needs pixel-level or caption-level semantic labels; CLIP is good at recognizing what is in an image but weak at where objects are, and vision foundation masks lack semantic labels.
Approach
Use unlabeled masks from SAM and DINO as supervision, and globally cluster mask features with online Sinkhorn-Knopp assignment against learnable class prompts fed through the CLIP text encoder, so the learned classes act as implicit semantic labels; a momentum image encoder stabilizes training.
Key results
- ConvNeXt-B with SAM masks improves over OpenCLIP baseline by about +16.2 mIoU average across benchmarks (e.g., ADE-150 13.1 to 20.3, CityScapes 16.2 to 34.8).
- PixelCLIP surpasses caption-supervised TCL and SegCLIP on all benchmarks and is competitive with SAM-CLIP, which uses 40M captions.
- Without global semantic clustering, performance collapses (e.g., COCO-Stuff 0.8 mIoU), showing clustering is critical.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP using unlabeled masks and global semantic clustering | CLIP ViT-B/16 · COCO-Stuff val | mIoU: 16.5 -> 23.6 | Helped |
| fine-tune CLIP using unlabeled masks and global semantic clustering | CLIP ViT-B/16 · ADE-150 val | mIoU: 13.2 -> 18.7 | Helped |
| fine-tune CLIP using unlabeled masks and global semantic clustering | CLIP ViT-B/16 · PASCAL-Context val | mIoU: 25.6 -> 37.9 | Helped |
| fine-tune CLIP using unlabeled masks and global semantic clustering | CLIP ViT-B/16 · CityScapes val | mIoU: 14.9 -> 27.2 | Helped |
| fine-tune CLIP using unlabeled masks and global semantic clustering | CLIP ViT-B/16 · PASCAL VOC val | mIoU: 73.9 -> 85.9 | Helped |
| add global semantic clustering of masks (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 0.8 -> 21.1 | Helped |
| add the CLIP text encoder (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 17.9 -> 21.1 | Helped |
| add learnable class prompts (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 18.2 -> 21.1 | Helped |
| add momentum encoder for mask features (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 19.4 -> 21.1 | Helped |
| increase cluster count from 64 to 256 (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 21.1 -> 21.3 | No clear effect |
| increase cluster count from 64 to 512 (ablation) | PixelCLIP · CityScapes val | mIoU: 33.2 -> 29.8 | Hurt |
| increase learnable prompt length from 1 to 4 (ablation) | PixelCLIP · COCO-Stuff val | mIoU: 20.2 -> 21.1 | Helped |
Surprising findings
- Removing global semantic clustering causes training to collapse and lose CLIP's pre-trained knowledge.
- Using COCO-Stuff ground-truth masks with class labels removed outperforms SA-1B SAM masks on most benchmarks.
Limitations
- Mask encoder initialized from CLIP weights gives poor mask classification early in training, causing noisy features and sub-optimal results.
- Mask pooling is used to extract per-mask CLIP features; alternative regional feature extractors are unexplored.
- Learned class prompts are not well interpreted; the authors leave deeper analysis of learned classes for future work.
Experimental setup
Datasets: SA-1B COCO-Stuff (masks only, ablation)
Benchmarks: COCO-Stuff ADE-20K (ADE-150) PASCAL-Context CityScapes PASCAL VOC zero-shot mask classification on ZegFormer and FC-CLIP masks
Metrics: mIoU
Baselines: CLIP OpenCLIP GroupViT CLIPpy OVSegmentor SegCLIP TCL SAM-CLIP ZeroSeg ZegFormer FC-CLIP
Builds on: CLIP SwAV-style online clustering CoOp prompt learning MaskCLIP SAM DINO
Closest papers by meaning
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.89)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.89)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.