Research map / Vision-language and multimodal / CLIP and zero-shot
Demystifying CLIP Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reverse-engineers CLIP's data curation via metadata-based sub-string matching and per-entry balancing (MetaCLIP), showing that curating CommonCrawl this way beats CLIP's WIT400M and LAION-400M under identical training budgets, scaling to 2.5B pairs for ViT-bigG at 82.1% ImageNet zero-shot.
Problem
CLIP's training data curation is undisclosed, and existing replications filter data with CLIP itself as a black-box teacher, which introduces biases and limits transparency and control over the data distribution.
Approach
Curate raw web image-text pairs without any model by matching texts against metadata entries (WordNet, Wikipedia unigrams/bigrams/titles) via sub-string matching and capping each entry's count at a threshold t via independent per-pair sampling, yielding a balanced data distribution over metadata.
Key results
- MetaCLIP 400M on ViT-B/32 reaches 70.8% ImageNet zero-shot with ViT-B/16 versus CLIP's 68.3%.
- Scaling to 2.5B pairs gives 79.2% on ViT-L/14, 80.5% on ViT-H/14, and 82.1% on ViT-bigG/14 ImageNet zero-shot.
- Training on the unbalanced 1.6B matched pool drops ImageNet accuracy to 61.9% vs 65.5% for balanced MetaCLIP 400M on ViT-B/32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use metadata-curated, balanced CommonCrawl data instead of CLIP's WIT400M data | ViT-B/16 · ImageNet zero-shot | accuracy: 68.3 -> 70.8 | Helped |
| Use metadata-curated, balanced CommonCrawl data instead of CLIP's WIT400M data | ViT-L/14 · ImageNet zero-shot | accuracy: 75.5 -> 76.2 | Helped |
| Scale curated training data from 400M to 1B | MetaCLIP ViT-L/14 · ImageNet zero-shot | accuracy: 76.2 -> 79.0 | Helped |
| Scale curated training data from 400M to 2.5B | MetaCLIP ViT-L/14 · ImageNet zero-shot | accuracy: 76.2 -> 79.2 | Helped |
| Change balancing threshold from t=20k to t=15k (ablation) | MetaCLIP ViT-B/32 · ImageNet zero-shot | accuracy: 65.5 -> 65.5 | No clear effect |
| Change balancing threshold from t=20k to t=35k (ablation) | MetaCLIP ViT-B/32 · Average accuracy across 26 tasks | accuracy: 58.2 -> 57.8 | Hurt |
| Remove balancing and train on the full matched pool (ablation) | MetaCLIP ViT-B/32 · ImageNet zero-shot | accuracy: 65.5 -> 61.9 | Hurt |
| Remove balancing and train on the full matched pool (ablation) | MetaCLIP ViT-B/32 · Average accuracy across 26 tasks | accuracy: 58.2 -> 56.6 | Hurt |
| Apply balancing online in the data loader (ablation) | MetaCLIP ViT-B/32 · Average accuracy across 26 tasks | accuracy: 58.2 -> 58.5 | Helped |
| Increase training data from MetaCLIP 1B to 2.5B | MetaCLIP ViT-L/14 · Average accuracy across 26 tasks | accuracy: 70.2 -> 69.8 | No clear effect |
Surprising findings
- Training on 4x more unbalanced matched data significantly hurts accuracy (61.9% vs 65.5% ImageNet).
- Applying the same curation to DataComp-12.8B's unfiltered pool yields a quality drop, suggesting implicit parser filters bias the pool.
Limitations
- Post-hoc curation relies on the metadata design choices (thresholds estimated, Wikipedia pageview sampling), which are not fully known for CLIP.
- Experiments use only the CLIP training recipe; results are conditioned on that fixed schedule and model architecture.
- Unfiltered DataComp pool showed quality drops attributed to suspected implicit parser biases, e.g., only http URLs kept.
Experimental setup
Datasets: CommonCrawl WordNet English Wikipedia DataComp-12.8B
Benchmarks: ImageNet 26 zero-shot classification tasks (CLIP eval suite) DataComp 38-task benchmark ImageNet distribution shift and VTAB and retrieval suite
Metrics: zero-shot top-1 accuracy average accuracy KL-divergence human rating
Baselines: CLIP (WIT400M) OpenCLIP (LAION-400M) DataComp-1B curated data
Builds on: CLIP OpenCLIP LAION-400M
Closest papers by meaning
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.93)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.92)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.91)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.91)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.