Research map / Detection and segmentation / Open-set and few-shot detection
Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models and rectifies space-preference and class-preference bias of frozen CLIP for unsupervised semantic segmentation via learnable Reference prompts, a positional-embedding projection, and logit subtraction, then distills into DeepLabV2.
Problem
CLIP applied to pixel-level unsupervised semantic segmentation exhibits spatial (center vs. boundary) and class-confusion biases that prior CLIP-based USS methods do not model, limiting segmentation accuracy.
Approach
Introduce a learnable "Reference" text prompt per class to capture class-preference bias and a convolutional projection of the ViT positional embedding to capture space-preference bias; subtract these bias logits from the fixed Query logits, train with a contrastive loss on Gumbel-Softmax masked features, then distill the rectified CLIP into DeepLabV2 with mask, feature, and text-guided losses.
Key results
- Ours (distill) reaches 75.4 mIoU on PASCAL VOC, outperforming CLIP-S4 (72.0) by 3.4 points.
- Rectification alone improves MaskCLIP from 49.5 to 58.5 mIoU on PASCAL VOC, and ADE20K from 9.5 to 11.1.
- Distillation reaches 33.8 mIoU on PASCAL Context and 14.3 on ADE20K, versus 33.6 and 13.5 (CLIPpy) for prior methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the rectified CLIP into a segmentation architecture | ReCLIP · PASCAL VOC 2012 | mIoU: 58.5 -> 75.4 | Helped |
| distill the rectified CLIP into a segmentation architecture | ReCLIP · PASCAL Context | mIoU: 25.8 -> 33.8 | Helped |
| distill the rectified CLIP into a segmentation architecture | ReCLIP · ADE20K | mIoU: 11.1 -> 14.3 | Helped |
| add learnable Reference logits (ablation) | CLIP direct segmentation · PASCAL VOC 2012 | mIoU: 49.5 -> 56.7 | Helped |
| add learnable Reference logits (ablation) | CLIP direct segmentation · PASCAL Context | mIoU: 21.7 -> 24.4 | Helped |
| add learnable Reference logits (ablation) | CLIP direct segmentation · ADE20K | mIoU: 9.5 -> 10.5 | Helped |
| add positional-embedding projection (ablation) | CLIP with learnable Reference logits · PASCAL VOC 2012 | mIoU: 56.7 -> 58.5 | Helped |
| add positional-embedding projection (ablation) | CLIP with learnable Reference logits · PASCAL Context | mIoU: 24.4 -> 25.8 | Helped |
| add positional-embedding projection (ablation) | CLIP with learnable Reference logits · ADE20K | mIoU: 10.5 -> 11.1 | Helped |
| subtract bias logits rather than add them (ablation) | ReCLIP bias rectification · PASCAL VOC 2012 | mIoU: 57.8 -> 58.5 | Helped |
| subtract bias logits rather than add them (ablation) | ReCLIP bias rectification · PASCAL Context | mIoU: 24.9 -> 25.8 | Helped |
| keep Query fixed rather than make it learnable (ablation) | ReCLIP prompt design · PASCAL Context and ADE20K | mIoU: 23.7 -> 24.4; 9.4 -> 10.5 | Helped |
Surprising findings
- Making the Query text embeddings learnable hurts performance, so only the Reference prompt is learned.
- Adding the bias logits instead of subtracting them drops VOC mIoU from 58.5 to 57.8, supporting subtraction as a useful prior.
Limitations
- Bias modeling is only evaluated on standard USS benchmarks with shared class sets between train and test.
- Distillation step adds a second training stage with DeepLabV2 on top of the rectified CLIP.
Experimental setup
Datasets: PASCAL VOC 2012 PASCAL Context ADE20K
Benchmarks: PASCAL VOC 2012 PASCAL Context ADE20K
Metrics: mIoU
Baselines: MaskCLIP MaskCLIP+ CLIPpy ReCo CLIP-S4 MaskContrast ACSeg HSG IIC SegSort Deep Spectral Methods
Builds on: MaskCLIP CoOp DeepLab V2 CLIP-S4
Closest papers by meaning
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.87)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.87)
- Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.86)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.86)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.