Research map / Detection and segmentation / Open-set and few-shot detection
Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation
Tong Shao, Zhuotao Tian, Hang Zhao, Jingyong Su
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic correlation recovery (self-correlation attention in last layer) to CLIP (ablation) | CLIP ViT-B/16 · Avg. of 4 benchmarks (224) | mIoU: 14.47 -> 36.94 | Improved |
| add patch clustering with DBSCAN on attention weights (ablation) | CLIPtrase (SCR) · Avg. of 4 benchmarks (224) | mIoU: 36.94 -> 38.02 | Improved |
| add denoising to discard global-patch noise clusters (ablation) | CLIPtrase (SCR+PC) · Avg. of 4 benchmarks (224) | mIoU: 38.02 -> 38.50 | Improved |
| use cosine similarity instead of inner product for correlation (ablation) | CLIPtrase · Avg. of 4 benchmarks (224) | mIoU: 25.91 -> 38.50 | Improved |
| cluster attention weight prototypes rather than features (ablation) | CLIPtrase · Avg. of 4 benchmarks (224) | mIoU: 35.61 -> 38.50 | Improved |
| apply correlation recovery at last layer rather than layers 9-11 (ablation) | CLIPtrase · Avg. of 4 benchmarks (224) | mIoU: 17.11 (layer 11) -> 38.50 (layer 12) | Improved |
| increase input resolution from 224 to 336 (ablation) | CLIPtrase · Avg. of 4 benchmarks | mIoU: 38.50 -> 39.31 | Improved |
| replace patch clustering masks with SAM using CLIPtrase point prompts | CLIPtrase · Avg. of 9 benchmarks | mIoU: 33.54 -> 34.49 | Improved |
| restore correlation within ground-truth regions (oracle) vs cosine over all patches (ablation) | CLIPtrase SCR · Avg. of VOC, ADE, COCO, PC (224) | mIoU: 36.94 -> 37.67 | Improved |
| use DINOv2 for masking instead of CLIPtrase clustering (ablation) | CLIPtrase · Avg. of VOC, ADE, COCO, PC (224) | mIoU: 38.50 -> 37.59 | Hurt |
| initialize SAN query embeddings with CLIPtrase features and masks | SAN · SAN training convergence | convergence speed | Improved |
Closest papers by meaning
- Enhancing Local Cognition of CLIP for Training-Free Open Vocabulary Semantic Segmentation (TPAMI 2026 · similarity 0.98)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.94)
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.91)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.91)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
