Research map / Label-efficient and robust learning / Recognition and long-tail
Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Classifier-guided CLIP Distillation (CCD), which refines CLIP pseudo-labels for unsupervised multi-label classification using CAM-guided local views from the trained classifier and a class-prior debiasing of CLIP predictions.
Problem
Unsupervised multi-label classification using CLIP pseudo-labels suffers from view-dependent predictions and class-level prediction bias of CLIP, which degrade the trained classifier's accuracy.
Approach
Select local views around objects using CAM from the classifier being trained (allocating more views to complex images), combine global and local CLIP pseudo-labels, debias CLIP probabilities by inverse class-wise average top-1 probability, and add a consistency loss between weakly and strongly augmented views.
Key results
- CCD reaches 90.1% mAP on VOC 2012 and 91.0% on VOC 2007, versus 88.6% and 89.0% for prior unsupervised SOTA CDUL.
- On MS COCO CCD achieves 70.3% mAP (CDUL 69.2%) and on NUS-WIDE 44.5% (CDUL 44.0%).
- Ablation on VOC 2012: classifier-guided label update adds 2.3 points over the 86.4% baseline; CLIP debiasing adds 0.7; combined with consistency loss reaches 90.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CCD instead of CDUL | Unsupervised multi-label classification · PASCAL VOC 2012 | mAP (%): 88.6 -> 90.1 | Helped |
| Use CCD instead of CDUL | Unsupervised multi-label classification · PASCAL VOC 2007 | mAP (%): 89.0 -> 91.0 | Helped |
| Use CCD instead of CDUL | Unsupervised multi-label classification · COCO | mAP (%): 69.2 -> 70.3 | Helped |
| Use CCD instead of CDUL | Unsupervised multi-label classification · NUSWIDE | mAP (%): 44.0 -> 44.5 | Helped |
| Add classifier-guided local label update (ablation) | CCD on PASCAL VOC 2012 · PASCAL VOC 2012 | mAP (%): 86.4 -> 88.7 | Helped |
| Add CLIP debias (ablation) | CCD with label update on PASCAL VOC 2012 · PASCAL VOC 2012 | mAP (%): 88.7 -> 89.4 | Helped |
| Add consistency loss (ablation) | CCD with label update on PASCAL VOC 2012 · PASCAL VOC 2012 | mAP (%): 88.7 -> 88.8 | Helped |
| Combine CLIP debias and consistency loss (ablation) | CCD with label update on PASCAL VOC 2012 · PASCAL VOC 2012 | mAP (%): 89.4 -> 90.1 | Helped |
| Add CLIP debias (ablation) | CCD on PASCAL VOC 2012, bottle class · PASCAL VOC 2012 bottle | AP (%): 77.0 -> 77.6 | Helped |
Surprising findings
- Consistency loss alone adds only 0.1 points, but combined with CLIP debiasing it yields a 1.4-point gain, suggesting debiasing amplifies noise that consistency loss suppresses.
- Using local pseudo-labels alone (weight 0) already outperforms CDUL, per the authors.
Limitations
- CAM-based local views struggle to separate co-occurring objects, producing noisy pseudo-labels on COCO.
- Gains are smaller on NUS-WIDE, attributed to 20% of images having no class objects and the simple prompt.
- Performance gap to fully supervised methods remains on COCO and NUS-WIDE.
Experimental setup
Datasets: PASCAL VOC 2012 PASCAL VOC 2007 MS COCO NUS-WIDE
Benchmarks: PASCAL VOC 2012 PASCAL VOC 2007 MS COCO NUS-WIDE
Metrics: mAP
Baselines: CDUL ROLE Naive AN Szegedy et al. Aodha et al. Durand et al. LL-R G2NetPL SARB ASL Chen et al. BCE
Builds on: CDUL CLIP Class Activation Mapping
Closest papers by meaning
- TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP without Training (AAAI 2024 · similarity 0.88)
- Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition (ICML 2024 · similarity 0.86)
- Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport (CVPR 2025 · similarity 0.86)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.86)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.85)
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.