Research map / Vision-language and multimodal / CLIP and zero-shot
Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives dense full scaling laws for open CLIP and MaMMUT models across three open datasets, uses them to compare architectures and datasets, and releases openMaMMUT-L/14 reaching 80.3% zero-shot ImageNet accuracy.
Problem
Comparing language-vision pre-training procedures and datasets at a single reference scale is unreliable because differences may stem from compute, scale, or data; scaling laws are needed for consistent comparison across scales, using fully open pipelines.
Approach
Use fitted scaling laws (with an irreducible-error term and held-out extrapolation checks) on dense measurements from open training pipelines to compare CLIP vs. MaMMUT, datasets, and other architectures, and show the comparison holds across tasks, datasets, and constant learning-rate schedules.
Key results
- MaMMUT crosses CLIP's scaling curve at roughly 1e10-1e11 GFLOPs and shows steeper scaling on ImageNet-1k zero-shot accuracy (alpha -0.233 vs -0.227 for CLIP, with a larger derivative at compute points).
- openMaMMUT-L/14 trained on 12.8B DataComp-1.4B samples reaches 80.3% zero-shot ImageNet-1k accuracy versus 79.2% reported for CLIP ViT-L/14 on the same data budget.
- Scaling laws show SigLIP has similar or worse scalability than CLIP on DataComp-1.4B, and DFN-1.4B gives the best scaling for both CLIP and MaMMUT on classification and retrieval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaMMUT rather than CLIP as compute increases | Language-vision pre-training on DataComp-1.4B, Re-LAION-1.4B, and DFN-1.4B · ImageNet-1k zero-shot classification at larger compute scales | error rate: crossing threshold between $10^{10}$ and $10^{11}$ GFLOPS | Helped |
| Use MaMMUT rather than CLIP as compute increases | Language-vision pre-training on DataComp-1.4B, Re-LAION-1.4B, and DFN-1.4B · MS-COCO image retrieval at larger compute scales | error rate (1-R@5): crossing threshold between $10^{10}$ and $10^{11}$ GFLOPS | Helped |
| Use MaMMUT rather than CLIP | ADE20K fine-tuned semantic segmentation · ADE20K | error rate (1-mIoU): $10^{9}$ GFLOPs | Helped |
| Train on DFN-1.4B rather than DataComp-1.4B or Re-LAION-1.4B | CLIP and MaMMUT · ImageNet-1k zero-shot classification | accuracy | Helped |
| Train on DFN-1.4B rather than DataComp-1.4B or Re-LAION-1.4B | CLIP and MaMMUT · MS-COCO retrieval | R@5 | Helped |
| Train on DataComp-1.4B rather than Re-LAION-1.4B | CLIP and MaMMUT · ImageNet-1k zero-shot classification | accuracy | Helped |
| Train on DataComp-1.4B rather than Re-LAION-1.4B | CLIP and MaMMUT · MS-COCO retrieval | R@5 | No clear effect |
| Replace CLIP softmax contrastive loss with SigLIP sigmoid loss | Language-vision model trained on DataComp-1.4B · ImageNet-1k zero-shot classification | scalability | No clear effect |
| Use decoder-only MaMMUT rather than encoder-decoder CoCa | Language-vision model trained on DataComp-1.4B · ImageNet-1k zero-shot classification | scalability | Helped |
| Combine contrastive and captioning losses rather than use captioning only | Language-vision model trained on DataComp-1.4B · ImageNet-1k zero-shot classification | scalability | Helped |
| Use captioning-only Cap rather than a contrastive model | Language-vision model trained on DataComp-1.4B · ImageNet-1k zero-shot classification | scalability | Hurt |
| Use a constant rather than cosine learning-rate schedule for scaling-law derivation | CLIP and MaMMUT on DataComp-1.4B · ImageNet-1k zero-shot classification and MS-COCO retrieval | scalability comparison: 98% less compute cost | No clear effect |
Surprising findings
- SigLIP shows similar or even worse scalability than CLIP under controlled data, contradicting claims of its architectural advantage.
- Scaling law fits with a constant learning rate schedule reproduce the same CLIP vs. MaMMUT crossover while saving about 98% of compute.
Limitations
- Mostly zero-shot evaluation; only segmentation uses fine-tuning, so linear probing and other transfer settings are not covered.
- Open datasets of 1.4B unique samples limit extrapolation to larger scales because of sample repetition beyond 1.4B; 12.8B-sample runs have about 9x repetition.
- Only contrastive and contrastive-plus-captioning objectives studied; other loss mixtures and architectural factors not covered; scaling-law derivation is computationally expensive.
Experimental setup
Datasets: DataComp-1.4B Re-LAION-1.4B DFN-1.4B ImageNet-1k MS-COCO ADE20K
Benchmarks: ImageNet-1k DataComp eval suite (35 tasks) ImageNet-V2 ImageNet robustness set MS-COCO ADE20K
Metrics: top-1 zero-shot accuracy R@5 1-mIoU mean squared error on held-out points 95% confidence intervals
Baselines: openCLIP OpenAI CLIP SigLIP CoCa Cap MetaCLIP EVA-CLIP DFN DataComp
Builds on: CLIP MaMMUT openCLIP Chinchilla-style compute-optimal scaling laws DataComp
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.93)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.91)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.91)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.89)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.