Research map / Vision-language and multimodal / CLIP and zero-shot
Classification Done Right for Vision-Language Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuperClass, a CLIP-free vision-language pretraining method that trains a ViT with a softmax classification loss over raw subword tokens (IDF-weighted, no filtering), pretrained on Datacomp-1B.
Problem
Contrastive vision-language pretraining like CLIP needs very large batch sizes and a text encoder; prior classification-based alternatives rely on hand-crafted text filtering and small-scale experiments, leaving their scalability unclear.
Approach
Use all subword tokens of each caption (from the CLIP BPE tokenizer) as a K-hot multi-label target, trained with a softmax cross-entropy over the vocabulary weighted by online IDF, so no text encoder, no contrastive batch, and no vocabulary curation are needed.
Key results
- SuperClass ViT-L/16 at 13B seen samples reaches 85.0% linear probing top-1 on ImageNet-1k vs 82.7% for OpenAI CLIP ViT-L/14 and 83.9% for OpenCLIP trained on Datacomp-1B.
- With LiT, SuperClass achieves 79.7% zero-shot ImageNet top-1, reported as better than OpenAI CLIP and OpenCLIP ViT-L/14 (which are not directly comparable).
- At 512M seen samples with ViT-L/16, SuperClass beats CLIP on all LLaVA-style tasks (e.g., VQAv2 70.40 vs 68.13, MMBench 67.72 vs 64.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace contrastive CLIP pretraining with SuperClass classification pretraining | ViT-Base · ImageNet-1K linear probing | top-1 accuracy (%): 78.5 -> 80.2 | Helped |
| Replace contrastive CLIP pretraining with SuperClass classification pretraining | ViT-Large · ImageNet-1K linear probing | top-1 accuracy (%): 82.7 -> 85.0 | Helped |
| Use a subword-level tokenizer instead of a word-level tokenizer (ablation) | SuperClass, ViT-L/16 · ImageNet-1K linear probing | accuracy: 80.3 -> 80.5 | Helped |
| Use a word-level tokenizer instead of a subword-level tokenizer (ablation) | SuperClass, ViT-S/16 · ImageNet-1K linear probing | accuracy: 67.9 -> 68.4 | Helped |
| Replace the OpenAI CLIP tokenizer with WordPiece (ablation) | SuperClass, ViT-L/16 · ImageNet-1K | linear probing accuracy: 80.5 -> 80.5 | No clear effect |
| Replace the OpenAI CLIP tokenizer with SentencePiece (ablation) | SuperClass, ViT-L/16 · ImageNet-1K | linear probing accuracy: 80.5 -> 80.2 | Hurt |
| Use Softmax loss instead of BCE, ASL, SoftMargin, or Two-way loss (ablation) | SuperClass, ViT-B/16 · ImageNet-1K | linear probing accuracy: 75.6 vs 73.6, 73.8, 73.5, 74.8 | Helped |
| Add IDF class weights (ablation) | SuperClass, ViT-B/16 · ImageNet-1K | linear probing accuracy: 75.6 -> 76.0 | Helped |
| Add IDF class weights (ablation) | SuperClass, ViT-B/16 · ImageNet-1K | zero-shot accuracy: 60.8 -> 61.7 | Helped |
| Keep stopwords in text labels instead of removing them (ablation) | SuperClass, ViT-B/16 · ImageNet-1K | linear probing accuracy: 75.7 -> 76.0 | Helped |
| Replace CLIP pretraining with SuperClass pretraining | ViT-L/16 · ImageNet-1K fine-tuning | top-1 accuracy: 86.5 -> 87.8 | Helped |
Surprising findings
- Plain softmax over K-hot text labels outperformed dedicated multi-label losses (BCE, ASL, soft margin, two-way loss) by a large margin.
- A word-level vocabulary (CatLIP-style) beat the subword tokenizer for the smallest ViT-S/16 model, while subword labels won at larger model sizes.
Limitations
- Ignores word order and object relationships in captions, losing supervisory information.
- Comparison with OpenAI CLIP is confounded by different training data (WIT-400M internal vs Datacomp-1B).
- Zero-shot results rely on LiT tuning and some comparisons are described as not directly comparable.
Experimental setup
Datasets: Datacomp-1B ImageNet-1k COCO captions LLaVA training data
Benchmarks: ImageNet-1k Pets Cars COCO captions VQAv2 GQA VizWiz T-VQA SQA MMBench MME POPE
Metrics: top-1 accuracy linear probing accuracy zero-shot accuracy 10-shot accuracy CIDEr VQA accuracy
Baselines: OpenAI CLIP OpenCLIP CatLIP DINOv2 MAE CapPa BEiT iBOT
Builds on: CLIP LiT CatLIP ClipCap LLaVA
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- Super-Class Guided Transformer for Zero-Shot Attribute Classification (AAAI 2025 · similarity 0.87)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.86)
- Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.