Research map / Vision-language and multimodal / CLIP and zero-shot
Reproducible Scaling Laws for Contrastive Language-Image Learning
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model, data, and samples-seen scale together | OpenCLIP trained on LAION · Zero-shot classification, retrieval, linear probing, and fine-tuning | downstream performance | Improved |
| Use LAION-400M/2B instead of WIT-400M pre-training | CLIP-style models · Zero-shot retrieval on MS-COCO and Flickr30K | scaling coefficient: MS-COCO: -0.08 vs -0.05; Flickr30K: -0.19 vs -0.10 | Mixed |
| Use WIT-400M instead of LAION-400M/2B pre-training | CLIP-style models · Zero-shot ImageNet and robustness classification | scaling coefficient: ImageNet: -0.16 vs -0.11; robustness: -0.24 vs -0.13 | Improved |
| Increase LAION data scale from 400M to 2B while keeping samples seen low | OpenCLIP ViT-B/32 and ViT-B/16 · Zero-shot classification | accuracy | No clear effect |
| Increase samples seen to 34B alongside the larger LAION-2B data scale | OpenCLIP ViT-B/32 and ViT-B/16 · Zero-shot classification | accuracy: 46.18 -> 50.69 | Improved |
| Increase samples seen from 13B to 34B on LAION-400M | OpenCLIP ViT-L/14 · MS-COCO and Flickr30K retrieval | Recall@5 | No clear effect |
| Switch larger-model training from float16 mixed precision to bfloat16 | OpenCLIP ViT-L/14, H/14, and g/14 · Pre-training stability | training stability | Improved |
| Use larger rather than smaller global batch sizes (ablation) | OpenCLIP · ImageNet zero-shot, LAION-400M, 13B samples | top-1 accuracy: 62.9 -> 63.37; 67.34 -> 67.86; 72.8 -> 72.98 | No clear effect |
| Replace LAION-400M with a random 400M subset of LAION-2B (ablation) | OpenCLIP ViT-B/32, 13B samples · ImageNet zero-shot | top-1 accuracy: 63.37 -> 63.56 | No clear effect |
| Replace full epoch-wise shuffling with resampling (ablation) | OpenCLIP ViT-B/32, LAION-400M, 13B samples · ImageNet zero-shot | top-1 accuracy: 63.28; 63.67 -> 63.37 | No clear effect |
| Use bfloat16 rather than float16 mixed precision (ablation) | Large OpenCLIP models · Pre-training loss stability | training stability | Improved |
Closest papers by meaning
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.95)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.89)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.89)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
