Research map / Label-efficient and robust learning / Recognition and long-tail
TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP without Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TagCLIP, a training-free local-to-global framework that turns frozen CLIP ViT-B/16 into an open-vocabulary multi-label tagger (patch-level classification, dual-masking attention refinement, class-wise reidentification) and uses its tags as pseudo labels for annotation-free segmentation (CLS-SEG).
Problem
CLIP's class token, trained with a contrastive softmax loss for global features, performs poorly on open-vocabulary multi-label classification because the global embedding is dominated by the most prominent class and misses small or inconspicuous objects.
Approach
Use patch tokens from the penultimate ViT layer (skipping the last self-attention, which destroys spatial information) for patch-level classification, refine scores with a dual-masked multi-layer attention affinity, and re-check each candidate class with a class-cropped global CLIP pass, all without training.
Key results
- TagCLIP reaches 92.8 mAP on PASCAL VOC and 68.8 mAP on COCO for multi-label classification, versus 85.8 and 63.3 for original CLIP, without any training.
- Ablation on VOC 2012 shows DMAR raises mAP from 85.4 to 93.9 and CWR adds a smaller gain to 94.1 with the full pipeline.
- CLS-SEG using TagCLIP tags as pseudo labels reaches 64.8 mIoU on VOC, 34.0 on COCO and 30.1 on COCO-Stuff, beating other annotation-free CLIP-based methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add patch-level classification, DMAR, and CWR to frozen CLIP | CLIP ViT-B/16 · PASCAL VOC 2007 test | mAP: 85.8 -> 92.8 | Helped |
| Add patch-level classification, DMAR, and CWR to frozen CLIP | CLIP ViT-B/16 · MS COCO 2014 val | mAP: 63.3 -> 68.8 | Helped |
| Omit the last self-attention operation (ablation) | CLIP ViT-B/16 · PASCAL VOC 2012 val | mIoU: 16.2 -> 41.6 | Helped |
| Add DMAR to coarse scores (ablation) | TagCLIP · PASCAL VOC 2012 val | mAP: 85.4 -> 93.9 | Helped |
| Add CWR to coarse scores (ablation) | TagCLIP · PASCAL VOC 2012 val | mAP: 85.4 -> 88.0 | Helped |
| Add CWR after DMAR (ablation) | TagCLIP · PASCAL VOC 2012 val | mAP: 93.9 -> 94.1 | Helped |
| Fuse multiple attention layers rather than using a single layer for refinement (ablation) | DMAR · Classification and segmentation benchmarks | classification and segmentation performance | Helped |
| Use dual masking in attention refinement (ablation) | DMAR · Classification and segmentation benchmarks | classification and segmentation performance | Helped |
| Fuse local and global classification scores (ablation) | TagCLIP CWR · PASCAL VOC 2007 test and MS COCO 2014 val | mAP | Helped |
| Add a background set (ablation) | TagCLIP · PASCAL VOC 2007 test | mAP: 92.4 -> 92.8 | Helped |
| Add a background set (ablation) | TagCLIP · MS COCO 2014 val | mAP: 67.9 -> 68.8 | Helped |
| Remove softmax from classification scores (ablation) | CLIP classification · PASCAL VOC 2007 test and MS COCO 2014 val | mAP: 85.8 -> 79.5; 63.3 -> 54.2 | Hurt |
Surprising findings
- Using raw logits without softmax causes a major performance drop for CLIP, suggesting the contrastive softmax is important for its scores.
- Skipping the last self-attention layer raises the segmentation mIoU of the final feature map from 16.2 to 41.6 on VOC 2012.
Limitations
- Performance is weaker for hierarchical categories with inclusion relations (e.g., cat and animal).
- Scene categories such as sky benefit less from discriminative local cues.
- Depends on a hand-chosen background word set and fixed thresholds (0.5).
Experimental setup
Datasets: PASCAL VOC 2012 MS COCO 2014 COCO-Stuff
Benchmarks: PASCAL VOC 2007 MS COCO 2014 PASCAL VOC 2012 MS COCO 2017 COCO-Stuff
Metrics: mAP mIoU
Baselines: CLIP DPT CLIPSurgery SARB DualCoOp TAI-DPT MaskCLIP GroupViT SegCLIP ReCo NamedMask
Builds on: CLIP CLIP-ES CLIP-Surgery GradCAM
Closest papers by meaning
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification (CVPR 2025 · similarity 0.88)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.87)
- Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.