AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Reproducible Scaling Laws for Contrastive Language-Image Learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev

CVPR 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 626 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Increase model, data, and samples-seen scale togetherOpenCLIP trained on LAION · Zero-shot classification, retrieval, linear probing, and fine-tuningdownstream performanceImproved
Use LAION-400M/2B instead of WIT-400M pre-trainingCLIP-style models · Zero-shot retrieval on MS-COCO and Flickr30Kscaling coefficient: MS-COCO: -0.08 vs -0.05; Flickr30K: -0.19 vs -0.10Mixed
Use WIT-400M instead of LAION-400M/2B pre-trainingCLIP-style models · Zero-shot ImageNet and robustness classificationscaling coefficient: ImageNet: -0.16 vs -0.11; robustness: -0.24 vs -0.13Improved
Increase LAION data scale from 400M to 2B while keeping samples seen lowOpenCLIP ViT-B/32 and ViT-B/16 · Zero-shot classificationaccuracyNo clear effect
Increase samples seen to 34B alongside the larger LAION-2B data scaleOpenCLIP ViT-B/32 and ViT-B/16 · Zero-shot classificationaccuracy: 46.18 -> 50.69Improved
Increase samples seen from 13B to 34B on LAION-400MOpenCLIP ViT-L/14 · MS-COCO and Flickr30K retrievalRecall@5No clear effect
Switch larger-model training from float16 mixed precision to bfloat16OpenCLIP ViT-L/14, H/14, and g/14 · Pre-training stabilitytraining stabilityImproved
Use larger rather than smaller global batch sizes (ablation)OpenCLIP · ImageNet zero-shot, LAION-400M, 13B samplestop-1 accuracy: 62.9 -> 63.37; 67.34 -> 67.86; 72.8 -> 72.98No clear effect
Replace LAION-400M with a random 400M subset of LAION-2B (ablation)OpenCLIP ViT-B/32, 13B samples · ImageNet zero-shottop-1 accuracy: 63.37 -> 63.56No clear effect
Replace full epoch-wise shuffling with resampling (ablation)OpenCLIP ViT-B/32, LAION-400M, 13B samples · ImageNet zero-shottop-1 accuracy: 63.28; 63.67 -> 63.37No clear effect
Use bfloat16 rather than float16 mixed precision (ablation)Large OpenCLIP models · Pre-training loss stabilitytraining stabilityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy