Research map / Detection and segmentation / Open-set and few-shot detection
Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPT-SEG, a one-stage CLIP-based zero-shot semantic segmentation method that adds FFT-based spectral prompts to shallow visual-encoder layers and a frequency-guided decoder, improving unseen-class mIoU on PASCAL VOC and COCO-Stuff.
Problem
Zero-shot semantic segmentation with CLIP: two-stage pipelines are expensive, and existing one-stage methods with visual prompt tuning do not fully exploit CLIP for pixel-level unseen-class segmentation, yielding incomplete segmentation.
Approach
Extends visual prompt tuning with learnable spectral prompts computed by FFT filtering of patch and CLS embeddings in the first two layers of CLIP's visual encoder, and adds a Spectral Guided Decoder that splits attention heads into high-frequency (local window attention) and low-frequency (average-pooled) branches to match text embeddings to pixels.
Key results
- On PASCAL VOC 2012, SPT-SEG reaches 87.4 unseen mIoU and 90.1 hIoU, versus 77.8 and 84.3 for ZegCLIP (the stated +9.6 unseen mIoU gain).
- On COCO-Stuff 164K, SPT-SEG reaches 40.6 seen mIoU, 43.8 unseen mIoU and 42.1 hIoU, versus ZegCLIP's 41.4 unseen mIoU and 40.8 hIoU.
- Ablation on VOC: SPT alone raises unseen mIoU to 86.7 and SGD alone to 79.9; combined they reach 87.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ZegCLIP with SPT-SEG | ZegCLIP · PASCAL VOC 2012 | mIoU(U): 77.8 -> 87.4 (+9.6) | Helped |
| replace ZegCLIP with SPT-SEG | ZegCLIP · COCO-Stuff 164K | mIoU(U): 41.4 -> 43.8 (+2.4) | Helped |
| add SPT (ablation) | ZegCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 | mIoU(U): 77.8 -> 86.7 | Helped |
| add SPT (ablation) | ZegCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 | hIoU: 84.3 -> 89.6 | Helped |
| add SGD (ablation) | ZegCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 | mIoU(U): 77.8 -> 79.9 | Helped |
| add SGD (ablation) | ZegCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 | hIoU: 84.3 -> 85.5 | Helped |
| add SPT and SGD (ablation) | ZegCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 | mIoU(U): 77.8 -> 87.4 | Helped |
| use SPT in layers 1-2 rather than layers 6-12 (ablation) | SPT-SEG · PASCAL VOC 2012 | mIoU(U): 80.9 -> 87.4 | Helped |
| use SPT in layers 1-2 rather than layers 11-12 (ablation) | SPT-SEG · PASCAL VOC 2012 | mIoU(U): 78.3 -> 87.4 | Helped |
| extend SPT from layers 1-2 to layers 1-12 (ablation) | SPT-SEG · PASCAL VOC 2012 | mIoU(U): 87.4 -> 86.5 | No clear effect |
| increase SGD depth from 1 to 3 layers (ablation) | SPT-SEG · PASCAL VOC 2012 | hIoU: 87.1 -> 90.1 | Helped |
| increase SGD depth from 3 to 5 layers (ablation) | SPT-SEG · PASCAL VOC 2012 | hIoU: 90.1 -> 87.7 | Hurt |
Surprising findings
- Applying spectral prompts only to the first two layers performs about as well as applying them across all 12 layers (hIoU 90.1 vs 89.4).
Limitations
- Pixel classification errors in complex scenes with occlusion and glass reflection, limited by CLIP recognition and resolution.
- Edge and fine-detail recognition still needs improvement.
- Ablation depth/layer results are on VOC only; the paper's reported gains are modest on COCO-Stuff.
Experimental setup
Datasets: PASCAL VOC 2012 COCO-Stuff 164K
Benchmarks: PASCAL VOC 2012 COCO-Stuff 164K
Metrics: pAcc mIoU(S) mIoU(U) hIoU
Baselines: SPNet ZS3 CaGNet SIGN Joint ZegFormer ZSSeg ZegCLIP
Builds on: CLIP Visual Prompt Tuning ZegCLIP LiTv2
Closest papers by meaning
- Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.88)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.88)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
- DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.88)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.