Research map / Detection and segmentation / Open-set and few-shot detection

Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation

CVPR 2024 · Detection and segmentation › Open-set and few-shot detection · approach: Zero-shot segmentation and detection · 25 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLIP-RC, a one-stage zero-shot semantic segmentation method that fine-tunes CLIP ViT-B/16 with a region-level bridge (masked attention region tokens), a region alignment module, and a training-only recovery decoder with recovery loss to curb overfitting.

Problem

Zero-shot semantic segmentation requires transferring CLIP's image-level knowledge to pixel-level prediction, while fine-tuning tends to overfit seen classes and erode CLIP's zero-shot generalization.

Approach

Use CLIP's strong region-level recognition via mask-attention region tokens (a region-level bridge, kept frozen) to provide fine-grained category features, fuse them with global and image features, and regularize tuning with a recovery decoder whose L1 recovery loss keeps features close to the original CLIP features.

Key results

• In the inductive setting on PASCAL VOC 2012, CLIP-RC reaches 88.4 hIoU, a 4.1-point gain over ZegCLIP (84.3).
• In the transductive setting on PASCAL VOC 2012 with self-training, CLIP-RC attains 93.0 hIoU, 1.9 points above the prior best.
• Ablation on PASCAL VOC 2012 shows the region-level bridge, region alignment module, and recovery decoder together raise hIoU from 84.3 to 88.4 over the ZegCLIP baseline.

Measured effects reported in the paper

Surprising findings

• Updating the region-level bridge tokens causes overfitting to seen classes and hurts unseen-class performance, so the bridge is kept frozen.

Limitations

• Finer region granularity improves IoU but increases computational load.
• Performance depends on the vision-language alignment quality of the CLIP pre-training.
• Two-stage and self-training variants are needed to reach the top transductive numbers.

Experimental setup

Datasets: PASCAL VOC 2012 PASCAL Context COCO-Stuff 164K

Benchmarks: PASCAL VOC 2012 PASCAL Context COCO-Stuff 164K

Metrics: pAcc mIoU(S) mIoU(U) hIoU

Baselines: SPNet ZS3 CaGNet SIGN Joint ZegFormer zsseg DeOP ZegCLIP MaskCLIP+ FreeSeg STRICT

Builds on: CLIP ZegCLIP VPT CoOp MMSegmentation

Closest papers by meaning

• Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation (AAAI 2024 · similarity 0.89)
• Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation (ICML 2024 · similarity 0.89)
• Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
• Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
• Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.87)
• Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.87)