Research map / Label-efficient and robust learning / Self-supervised and contrastive
What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled ResNet-50 study of why CLIP is more robust than supervised learning to long-tailed web pre-training data, attributing it to dynamic (subsampled-vocabulary) classification, descriptive language supervision, and data scale/open-world concepts, and transferring the trick to supervised and DINO self-supervised training.
Problem
CLIP is notably more robust than supervised learning to the severe class imbalance present in web-scale image-text data, but the mechanisms behind this robustness are not understood.
Approach
Controlled experiments on ImageNet-Captions and LAIONet show CLIP's contrastive pretext task forms a dynamic classification problem with a small per-batch vocabulary that isolates head-class bias; reproducing this by randomly subsampling the classifier vocabulary in supervised learning (and frozen CLIP text prototypes) yields CLIP-like tail generalization.
Key results
- Vocabulary subsampling in supervised ResNet-50 training reduces prediction bias to a level similar to template-based CLIP, with a preferred vocabulary size of around 100 on ImageNet-Captions.
- Descriptive language supervision and data scale both improve CLIP's robustness and discriminability together; a frozen pre-trained CLIP text head gives better data efficiency and robustness than training from scratch or fine-tuning.
- DINO trained on LAIONet with prototype subsampling narrows the transfer gap to ImageNet-pretrained DINO across most of 11 of 12 transfer datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase natural-language caption descriptiveness | CLIP · ImageNet-Captions | ImageNet accuracy | Helped |
| Use less descriptive or template-based text supervision instead of more descriptive captions (ablation) | CLIP · ImageNet-Captions | robustness and discriminability | Hurt |
| Subsample the training vocabulary to form dynamic classification tasks | supervised learning with ResNet-50 · ImageNet-Captions | prediction bias | Helped |
| Use an intermediate vocabulary size, around 100 (ablation) | supervised learning with ResNet-50 · ImageNet-Captions | accuracy | Helped |
| Increase data imbalance to an extreme level (ablation) | CLIP · LAIONet | prediction bias | Hurt |
| Use web-data distribution shift with a pre-trained text head (ablation) | CLIP · LAIONet | robustness to data imbalance | Helped |
| Use web-data distribution shift (ablation) | CLIP · LAIONet | discriminability | Hurt |
| Increase intra-class data diversity by using a smaller filtering threshold (ablation) | CLIP · LAIONet | robustness to data imbalance | Helped |
| Scale up pre-training data (ablation) | CLIP · LAIONet | ImageNet zero-shot accuracy | Helped |
| Initialize with a frozen pre-trained CLIP text encoder (ablation) | CLIP · LAIONet | robustness to data imbalance | Helped |
| Fine-tune the pre-trained CLIP text encoder instead of freezing it (ablation) | CLIP · LAIONet | intra-class variation | Hurt |
| Use a frozen RoBERTa text encoder for the contrastive task (ablation) | CLIP · LAIONet | performance | Hurt |
Surprising findings
- Frozen CLIP class prototypes alone do not give tail generalization for supervised learning; vocabulary subsampling is necessary to match CLIP.
- Fine-tuning the CLIP text head causes over-fitting, and a RoBERTa text head adversarially affects performance.
Limitations
- Class frequency estimation relies on a rule-based caption matching pipeline prone to noise and ambiguity.
- Study covers classification-only transfer; open-world detection and segmentation not studied.
- CLIP remains biased toward tail classes on average despite robustness gains.
Experimental setup
Datasets: ImageNet-Captions LAIONet YFCC-15M CC-12M LAION-400M MetaCLIP-400M
Benchmarks: ImageNet ImageNet-Captions-100 DINO transfer benchmark (12 datasets)
Metrics: Spearman correlation between class frequency and per-class a top-1 accuracy neural collapse metrics NC1 and NC2 linear probing accuracy
Baselines: supervised learning (ResNet-50) template-based CLIP vanilla DINO RoBERTa text head CLIP zero-shot
Builds on: CLIP DINO neural collapse federated loss
Closest papers by meaning
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.88)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.88)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.88)
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Does CLIP’s generalization performance mainly stem from high train-test similarity? (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.