Research map / Vision-language and multimodal / CLIP and zero-shot
SuperCLIP: CLIP with Simple Classification Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a lightweight linear classification head on the vision encoder that predicts IDF-weighted bag-of-subword-token labels from captions, trained jointly with CLIP's contrastive loss, improving zero-shot, retrieval, and visual tasks at ~0.077% extra FLOPs.
Problem
CLIP's global contrastive objective underuses fine-grained semantic signals in text (attributes, relations, actions), and this worsens with long re-captioned texts and small batch sizes.
Approach
Treat raw caption subword tokens as multi-label classification targets for the vision encoder via a linear head on pooled image features, with IDF weighting, and add this cross-entropy loss to the CLIP objective. The loss does not depend on batch size and needs no extra annotations.
Key results
- SuperCLIP-L at 512M samples improves ImageNet val zero-shot from 66.1 to 70.1 and COCO image retrieval from 32.7 to 35.9 over CLIP-L.
- Under mixed Recap-DataComp captions (Dual setting), SuperCLIP-L reaches 49.5 average zero-shot accuracy over 38 datasets versus 46.8 for CLIP-L (0.8/0.2 mix).
- The linear head and classification loss add only 0.077% to total FLOPs for the L-size model, and SuperCLIP mitigates CLIP's drop at small batch sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-classification supervision | CLIP ViT-B, B-512M · ImageNet-1K val | accuracy: 60.5 -> 63.5 | Helped |
| Add token-classification supervision | CLIP ViT-B, B-512M · COCO image retrieval | Recall@1: 29.0 -> 31.3 | Helped |
| Add token-classification supervision | CLIP ViT-L, L-512M · ImageNet-1K v2 | accuracy: 57.4 -> 62.5 | Helped |
| Add token-classification supervision | CLIP ViT-B, B-512M · PASCAL | mIoU: 57.8 -> 65.5 | Helped |
| Add token-classification supervision | CLIP ViT-B, B-512M · NYUv2 | RMSE: 0.768 -> 0.746 | Helped |
| Use IDF weighting for token labels instead of unweighted K-hot labels (ablation) | SuperCLIP ViT-B/16 · ImageNet-1K val | accuracy: 44.8 -> 47.1 | Helped |
| Use IDF weighting for token labels instead of unweighted K-hot labels (ablation) | SuperCLIP ViT-B/16 · COCO image retrieval | Recall@1: 31.6 -> 33.2 | Helped |
| Increase classification-loss weight from 0.4 to 1.0 (ablation) | SuperCLIP ViT-B/16 · ImageNet-1K val | accuracy: 44.1 -> 47.1 | Helped |
| Increase classification-loss weight from 1.0 to 1.6 (ablation) | SuperCLIP ViT-B/16 · ImageNet-1K val | accuracy: 47.1 -> 47.2 | No clear effect |
| Train with rich long captions using classification supervision | SuperCLIP-B, long-caption-only training · Average zero-shot classification across 38 datasets | accuracy: 27.8 -> 31.4 | Helped |
| Add token-classification supervision to SigLIP | SigLIP B-512M · ImageNet-1K val | accuracy: 60.4 -> 64.1 | Helped |
| Add token-classification supervision to FLIP | FLIP B-512M · Flickr30K text retrieval | Recall@1: 66.7 -> 72.0 | Helped |
Surprising findings
- Replacing all DataComp captions with re-captioned long captions lowers CLIP's performance, while adding the classification loss recovers much of the gain from long captions.
Limitations
- Supervision only flows from text to the vision encoder; the text encoder is not improved.
- Evaluation of many results is on a single-setting ablation (256M samples, ViT-B/16) and a limited set of baselines.
Experimental setup
Datasets: DataComp-1B Recap-DataComp-1B ImageNet-1K COCO Flickr30K ADE20K PASCAL NYUv2 VQAv2 GQA MMBench
Benchmarks: ImageNet-1K COCO Flickr30K 38 zero-shot datasets (LAION CLIP Benchmark) PASCAL ADE20K NYUv2 VQAv2 GQA VizWiz TextVQA SQA
Metrics: zero-shot accuracy Recall@1 linear probing accuracy mIoU RMSE TFLOPs/GFLOPs
Baselines: CLIP SigLIP FLIP Long-CLIP SigLIP 2 Recap-DataComp-1B
Builds on: CLIP SigLIP FLIP LLaVA-1.5
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.92)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.91)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.91)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.91)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.91)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.