Research map / Detection and segmentation / Open-set and few-shot detection
Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPSeg, a training-free open-vocabulary segmentation method that builds attention from middle-layer CLIP keys (coherence-enhanced residual attention) and values from integrated deep-layer dense outputs (deep semantic integration).
Problem
Training-free open-vocabulary semantic segmentation with frozen CLIP suffers because the final layer's features prioritize global over local representations, giving noisy attention and value embeddings contaminated by dominant categories.
Approach
Uses key features from middle layers (chosen by an intra-image coherence metric, AUROC on COCO-Stuff) to build self-self attention with a relation-attention term, and combines the final value with dense outputs regenerated from deep layers (excluding self-attention) to form the value embeddings.
Key results
- With CLIP ViT-L, CLIPSeg reaches 35.7 average mIoU with PAMR, a 7.8-point gain over ClearCLIP (27.9).
- With MetaCLIP ViT-L-5B, CLIPSeg reaches 37.4 average mIoU, 8.9 points above GEM (28.5).
- With CLIP ViT-B, CLIPSeg gets 11.5 mIoU on C459 training-free, above DeOP's 9.4 trained on COCO-Stuff.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ClearCLIP with CLIPSeg | CLIP-L · Average across 9 benchmarks | mIoU (%): 27.9 -> 35.7 (+7.8) | Helped |
| use the optimal middle-layer key embedding kM instead of final-layer kL for attention (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 24.2 -> 29.6 | Helped |
| add relation-attention G(kM) to kM-kM attention (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 29.6 -> 30.1 | Helped |
| use the optimal middle-layer key embedding kM instead of final-layer kL for attention (ablation) | CLIP-L · Average across 5 benchmarks | mIoU (%): 26.2 -> 32.6 | Helped |
| add relation-attention G(kM) to kM-kM attention (ablation) | CLIP-L · Average across 5 benchmarks | mIoU (%): 32.6 -> 32.9 | Helped |
| combine dense outputs from the last three layers with final value vL instead of using vL alone (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 39.8 -> 40.7 | Helped |
| combine dense outputs from the last three layers with final value vL instead of using vL alone (ablation) | Meta-L-400M · Average across 5 benchmarks | mIoU (%): 41.8 -> 43.3 | Helped |
| set temperature τ from 5 to 6 (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 39.6 -> 40.7 | Helped |
| set temperature τ from 6 to 7 (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 40.7 -> 40.7 | No clear effect |
| set temperature τ from 7 to 8 (ablation) | CLIP-B · Average across 5 benchmarks | mIoU (%): 40.7 -> 40.3 | Hurt |
| set temperature τ from 5 to 6 (ablation) | CLIP-L · Average across 5 benchmarks | mIoU (%): 42.7 -> 43.4 | Helped |
| set temperature τ from 6 to 7 (ablation) | CLIP-L · Average across 5 benchmarks | mIoU (%): 43.4 -> 43.3 | Hurt |
Surprising findings
- Existing training-free methods perform worse with ViT-L than ViT-B CLIP models, which the authors attribute to poor final-layer feature coherence.
Limitations
- Hyperparameters such as the temperature and the cosine threshold are set empirically and vary in their optimal values across datasets.
- The middle layer used for attention must be chosen per model from a coherence analysis (Fig. 4) rather than automatically.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: Pascal VOC20 Pascal Context 59 Pascal Context 60 COCO-Stuff COCO-Object ADE20k A150 ADE20k A847 Cityscapes
Metrics: mIoU
Baselines: MaskCLIP SCLIP GEM CLIPtrase ClearCLIP Zegformer OVSeg DeOP SegCLIP TCL CoDe
Builds on: CLIP MaskCLIP ClearCLIP CLIP-Surgery GEM STEGO PAMR
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.90)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.89)
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.89)
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.