Research map / Architectures and efficiency / RNNs and normalization
Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that optimal data filtering for CLIP-style VLM training depends on compute because repeated high-quality data loses utility, and proposes repetition-aware scaling laws that predict performance on data pool mixtures from individual pool parameters.
Problem
Data filtering for vision-language pretraining is typically chosen independently of training compute, yet repeated high-quality data loses utility, so the optimal filtering threshold may shift with compute budget.
Approach
Extends neural scaling laws to heterogeneous web data by modeling per-sample utility decay under repetition (exponential half-life) and predicting mixture-of-pool performance from individual pool scaling parameters, without training on the mixtures.
Key results
- Beyond about 450M samples seen, training on unfiltered common crawl outperforms LAION-filtered (10%) data on the 18-task average.
- At 128M training iterations, LAION-style filtering beats no filtering by 7.5% zero-shot accuracy averaged over 18 tasks.
- Estimated optimal filter shifts from Top 10% at under 100M samples seen to Top 20% at 100M-350M and Top 30% beyond 350M samples seen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply LAION CLIP-score filtering (retain 10%) instead of no filtering | ViT-B/32 CLIP trained on Common Crawl · DataComp medium; average accuracy over 18 tasks, 128M training iterations | average zero-shot accuracy: +7.5% | Helped |
| apply LAION CLIP-score filtering (retain 10%) instead of no filtering | ViT-B/32 CLIP trained on Common Crawl · DataComp medium; beyond 450M training iterations | average zero-shot accuracy | Hurt |
| retain only the top 10–20% by CLIP score rather than top 40% (ablation) | CLIP score filtering · DataComp medium; 32M compute | average zero-shot accuracy | Helped |
| retain top 40% by CLIP score rather than top 10–20% (ablation) | CLIP score filtering · DataComp medium; 32M compute | average zero-shot accuracy | Hurt |
| retain more data instead of only the top 10% by CLIP score (ablation) | CLIP score filtering · DataComp medium; 640M compute | average zero-shot accuracy | Helped |
| retain more data rather than only the top 20% by T-MARS score (ablation) | T-MARS filtering · DataComp medium; increasing compute from 32M to 640M | average zero-shot accuracy | Helped |
| use the top 10% T-MARS pool instead of a larger, lower-quality pool (ablation) | DataComp medium · ImageNet zero-shot; up to 100M samples seen | zero-shot performance | Helped |
| use top 20% rather than top 10% T-MARS data (ablation) | DataComp medium · ImageNet zero-shot; 100M to 350M samples seen | zero-shot performance | Helped |
| use top 30% rather than top 20% T-MARS data (ablation) | DataComp medium · ImageNet zero-shot; beyond 350M samples seen | zero-shot performance | Helped |
| repeat the highest-quality data pool rather than use lower-quality data (ablation) | DataComp medium T-MARS-ranked pools · DataComp medium; after four epochs | instantaneous utility | Hurt |
Surprising findings
- Highest-quality pool (Top 10%) drops below the lowest-quality pool (Top 30%-40%) in utility after the fourth epoch.
- Unfiltered common crawl outperforms LAION-filtered data at high compute, reversing the low-compute ordering.
Limitations
- Fixed batch size of 4096; scaling laws do not account for batch size effects.
- Unclear whether scaling parameters stay consistent when pool size is scaled by orders of magnitude.
- Assumes data diversity (half-life) stays consistent when pools are mixed.
Experimental setup
Datasets: DataComp medium pool (128M) LAION-filtered subset
Benchmarks: ImageNet zero-shot DataComp 18-task average
Metrics: zero-shot accuracy prediction error of scaling curves
Baselines: LAION filtering (CLIP score top 10%) T-MARS no filtering (raw common crawl) Cherti et al. scaling law
Builds on: Kaplan scaling laws Muennighoff data-constrained scaling DataComp T-MARS Cherti et al. CLIP scaling laws
Closest papers by meaning
- A Bitter Lesson for Data Filtering (NeurIPS 2026 · similarity 0.89)
- Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality (NeurIPS 2025 · similarity 0.89)
- Data Filtering Networks (ICLR 2024 · similarity 0.88)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.88)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.86)
- DataRater: Meta-Learned Dataset Curation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.