Research map / Detection and segmentation / Open-set and few-shot detection
ZegCLIP: Towards Adapting CLIP for Zero-Shot Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deep prompt tuning instead of fine-tuning the CLIP image encoder (ablation) | ZegCLIP · PASCAL VOC 2012 | mIoU(U): 13.8 -> 28.3 | Helped |
| Use non-mutually exclusive loss rather than the exclusive-loss baseline (ablation) | Baseline-Fix · PASCAL VOC 2012 | mIoU(U): 16.3 -> 36.6 | Helped |
| Add a Relationship Descriptor to the baseline (ablation) | Baseline-Fix · PASCAL VOC 2012 | mIoU(U): 16.3 -> 46.6 | Helped |
| Add both NEL and RD (ablation) | Baseline-DPT · PASCAL VOC 2012 | mIoU(U): 28.3 -> 77.8 | Helped |
| Replace DPT with fine-tuning while retaining NEL and RD (ablation) | ZegCLIP · PASCAL VOC 2012 | mIoU(U): 77.8 -> 42.4 | Hurt |
| Add focal loss and dice loss to BCE (ablation) | ZegCLIP · PASCAL VOC 2012 | hIoU: 80.9 -> 84.3 | Helped |
| Add focal loss and dice loss to BCE (ablation) | ZegCLIP · COCO-Stuff 164K | hIoU: 38.9 -> 40.8 | Helped |
| Use [t⊙g, t] rather than text embedding t alone as the query (ablation) | ZegCLIP · VOC | mIoU(U): 33.7 -> 77.8 | Helped |
| Use text-image sum t+g as the query (ablation) | ZegCLIP · VOC | mIoU(U): 33.7 -> 13.9 | Hurt |
| Apply self-training (ablation) | ZegCLIP · COCO-Stuff 164K, transductive | mIoU(U): 41.4 -> 54.8 | Helped |
| Use one-stage ZegCLIP instead of two-stage ZegFormer | zero-shot segmentation inference on VOC · VOC | FPS: 1.7 -> 9.0 | Helped |
Closest papers by meaning
- Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation (CVPR 2024 · similarity 0.92)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.91)
- Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation (ICML 2024 · similarity 0.90)
- Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation (AAAI 2024 · similarity 0.90)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.90)
- DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.