Research map / Vision-language and multimodal / CLIP and zero-shot
Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExCEL, which explores CLIP's dense patch-text alignment for weakly supervised semantic segmentation using LLM-generated attribute knowledge and a frozen-CLIP visual calibration with a lightweight adapter.
Problem
Weakly supervised semantic segmentation from image-level labels relies on CAMs that highlight only discriminative parts; existing CLIP-based WSSS methods use global image-text alignment and leave CLIP's dense patch-text alignment unexplored.
Approach
Generates CAMs via patch-wise cosine similarity between CLIP patch features and text embeddings, enriched by LLM-generated descriptions clustered into implicit attributes (TSE), and calibrates frozen CLIP features via a non-parametric intra-correlation across intermediate layers (SVC) plus a learnable distribution-shift adapter supervised by a diversity loss (LVC).
Key results
- ExCEL achieves 78.4% mIoU on VOC val and 78.5% on test, and 50.3% on COCO val, versus WeCLIP's 76.4%/77.2%/47.1%.
- Training-free ExCEL reaches 74.6% CAM mIoU on VOC train, outperforming CLIP-ES's 70.8%; with LVC it reaches 78.0%.
- ExCEL uses 3.2 GB GPU memory and 90 minutes of training, about 6% of MCTformer+ training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ExCEL instead of WeCLIP | CLIP ViT-B WSSS · PASCAL VOC val | mIoU: 76.4 -> 78.4 | Helped |
| use ExCEL instead of WeCLIP | CLIP ViT-B WSSS · PASCAL VOC test | mIoU: 77.2 -> 78.5 | Helped |
| use ExCEL instead of WeCLIP | CLIP ViT-B WSSS · MS COCO val | mIoU: 47.1 -> 50.3 | Helped |
| add SVC (ablation) | CLIP baseline · PASCAL VOC val | mIoU: 12.1 -> 72.5 | Helped |
| add TSE to the SVC configuration (ablation) | CLIP ViT-B WSSS · PASCAL VOC val | mIoU: 72.5 -> 75.1 | Helped |
| add LVC to the SVC and TSE configuration (ablation) | CLIP ViT-B WSSS · PASCAL VOC val | mIoU: 74.7 -> 77.2 | Helped |
| replace attribute clustering with explicit fusion of descriptions (ablation) | ExCEL TSE · PASCAL VOC val | mIoU: 77.2 -> 75.1 | Hurt |
| vary the number of attributes from 112 to 144 (ablation) | ExCEL TSE · PASCAL VOC val | mIoU: 77.2 -> 77.0 | Hurt |
| vary the number of attributes from 112 to 196 (ablation) | ExCEL TSE · PASCAL VOC val | mIoU: 77.2 -> 76.5 | Hurt |
| replace q-k attention with Intra-correlation (ablation) | CLIP ViT-B visual features · PASCAL VOC train | mIoU: 11.2 -> 69.7 | Helped |
| apply Intra-correlation across intermediate layers instead of only the last layer (ablation) | ExCEL SVC · PASCAL VOC train | mIoU: 69.7 -> 74.6 | Helped |
| add LVC to SVC (ablation) | ExCEL · PASCAL VOC train | mIoU: 74.6 -> 78.0 | Helped |
Surprising findings
- Training-free ExCEL with frozen CLIP produces CAMs of 74.6% mIoU on VOC train, comparable to many training-required WSSS methods.
Limitations
- Relies on GPT-4 to generate per-class descriptions, adding dependence on a proprietary LLM.
- Hyperparameters (B attributes, N layers, scaling factors) are tuned per dataset.
Experimental setup
Datasets: PASCAL VOC 2012 MS COCO 2014 GPT-4 generated class descriptions
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU precision recall
Baselines: CLIP-ES WeCLIP CLIMS DIAL DuPL SeCo ToCo CPAL
Builds on: CLIP CLIP-ES WeCLIP MaskCLIP DenseCLIP
Closest papers by meaning
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP (AAAI 2025 · similarity 0.89)
- Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.88)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.88)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.