Research map / Detection and segmentation / Weakly supervised detection and segmentation
Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WeCLIP, a single-stage weakly supervised semantic segmentation pipeline that uses a frozen CLIP ViT-B backbone with a light transformer decoder and a refinement module (RFM) that dynamically refines GradCAM pseudo labels using the decoder's affinity map.
Problem
Weakly supervised semantic segmentation with image-level labels: existing single-stage methods use trainable ImageNet backbones and underperform multi-stage methods, while CLIP is used only to generate offline pseudo labels rather than as a segmentation backbone.
Approach
Use a frozen CLIP image encoder directly as the segmentation backbone with a small learnable decoder over multi-layer features, and refine the static CAM pseudo labels during training via an affinity map learned by the decoder, so decoder and refinement mutually improve.
Key results
- WeCLIP reaches 76.4% mIoU on VOC 2012 val and 77.2% on test with DenseCRF, outperforming the prior best single-stage ToCo by 5.3% and 5.0%.
- WeCLIP achieves 47.1% mIoU on MS COCO-2014 val, more than 4.8% above single-stage methods and above the multi-stage CLIP-ES (45.4%).
- Ablation: the RFM adds 6.2% mIoU (68.7% to 74.9%) on VOC val; training uses 6.2GB GPU memory and under 6M learnable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RFM to the decoder (ablation) | WeCLIP on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 68.7 -> 74.9 | Helped |
| Use 3 rather than 1 or 2 transformer layers in the decoder (ablation) | WeCLIP on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 73.2; 74.4; 74.9 | Helped |
| Use 4 or 5 rather than 3 transformer layers in the decoder (ablation) | WeCLIP on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 74.9 -> 72.6; 70.3 | Hurt |
| Add Ge filtering to CLIP attention maps (ablation) | WeCLIP CAM refinement on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 71.8 -> 72.3 | Helped |
| Add the decoder affinity map to CLIP attention maps (ablation) | WeCLIP CAM refinement on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 71.8 -> 74.3 | Helped |
| Combine affinity map, Ge filtering, and CLIP attention maps (ablation) | WeCLIP CAM refinement on PASCAL VOC 2012 val · PASCAL VOC 2012 val | mIoU (%): 74.3 -> 74.9 | Helped |
| Apply CRF post-processing | WeCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 val and test | mIoU (%): 74.9 -> 76.4; 75.2 -> 77.2 | Helped |
| Apply CRF post-processing | WeCLIP on MS COCO-2014 val · MS COCO-2014 val | mIoU (%): 46.4 -> 47.1 | Helped |
| Use WeCLIP rather than CLIP-ES | Weakly supervised segmentation on PASCAL VOC 2012 · PASCAL VOC 2012 val and test | mIoU (%): 73.8 -> 76.4; 73.9 -> 77.2 | Helped |
| Use WeCLIP rather than CDL | Weakly supervised segmentation on MS COCO-2014 · MS COCO-2014 val | mIoU (%): 45.5 -> 47.1 | Helped |
Surprising findings
- A frozen CLIP backbone with no fine-tuning, plus a light decoder, outperforms fine-tuned ImageNet-pretrained single-stage and many multi-stage methods on VOC and COCO.
Limitations
- Frozen backbone provides static CAM, so the refinement relies on the learned decoder affinity to correct errors.
- Relies on DenseCRF post-processing for the best reported numbers (76.4 vs 74.9 without CRF on VOC val).
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO-2014
Benchmarks: PASCAL VOC 2012 MS COCO-2014
Metrics: mIoU
Baselines: ToCo CLIP-ES AFA TSCD MCTformer RIB CLIMS SLRNet
Builds on: CLIP-ES AFA ToCo CLIMS
Closest papers by meaning
- CLIP Is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation (CVPR 2023 · similarity 0.92)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.89)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation (ICML 2024 · similarity 0.87)
- Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.