Research map / Large language models / Language, safety and interpretability
DataDecide: How to Predict Best Pretraining Data with Small Experiments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains a 1,050-model suite (25 pretraining data recipes x 14 scales from 4M to 1B params x 3 seeds, 100 tokens/param) and benchmarks how well small-scale experiments and scaling-law fits predict which data recipe wins at 1B scale.
Problem
Choosing pretraining datasets with small-scale experiments is cheap but its decision accuracy at the target scale is unvalidated; it is unclear which benchmarks, prediction methods, and metrics most reliably pick the best data for large models.
Approach
Measure prediction quality by pairwise decision accuracy (whether the small-scale ranking picks the correct winner at 1B, averaged over 3 seeds) across many controlled data recipes, and show that single-scale ranking is a strong baseline while continuous likelihood proxies (Correct Prob) make hard benchmarks such as code tasks predictable.
Key results
- Ranking models at a single small size (150M) predicts the best 1B data recipe about 80% of pairwise comparisons correctly.
- None of the 8 multi-scale scaling-law variants exceeds the compute-decision frontier of single-scale ranking.
- Using the continuous Correct Prob proxy makes MBPP and HumanEval go from trivial to roughly 80% decision accuracy at small scale, and MMLU, ARC, HellaSwag exceed 80% predictability at 0.01% of compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rank data recipes at a single small model scale instead of extrapolating scaling laws | Data recipe selection for 1B models · Pairwise data recipe comparisons | decision accuracy: ~80% | Helped |
| Use continuous likelihood proxies, including Correct Prob or Total Prob, instead of Accuracy in small experiments | Small-scale prediction of target-scale OLMES Accuracy · OLMES tasks | decision accuracy: >80% | Helped |
| Add a helper point to the 3-parameter fit (ablation) | 3-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 6.0 | Helped |
| Exclude the first 50% of intermediate checkpoints from the second-stage fit (ablation) | 3-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 5.9 | Helped |
| Add helper points and exclude the first 50% of intermediate checkpoints (ablation) | 3-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 5.6 | Helped |
| Remove the irreducible loss term E (ablation) | 3-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 6.5 | No clear effect |
| Replace the two-step fit with a single-step compute-to-accuracy fit (ablation) | 3-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 42.9 | Hurt |
| Replace the two-step fit with a single-step fit using parameter and token counts (ablation) | 5-parameter scaling law · OLMES Accuracy; 1B targets | relative error: 6.5 -> 42.8 | Hurt |
| Model loss using parameter count and token count directly instead of compute (ablation) | Scaling law prediction · OLMES Accuracy; 1B targets | relative error: 6.5 -> 230.8 | Hurt |
| Use scaling-law extrapolation rather than ranking single-scale experiments | Data recipe selection · OLMES tasks | decision accuracy | No clear effect |
Surprising findings
- Single-scale ranking of data recipes is a strong baseline that is hard for scaling-law methods to beat.
- Code tasks (HumanEval, MBPP) move from near-chance decision accuracy to about 80% when switching from discrete accuracy to a continuous likelihood proxy.
Limitations
- Only one token-to-parameter ratio (100, 5x Chinchilla) is studied.
- Only 14 model sizes from 4M to 1B parameters and 25 data recipes; findings may not generalize to other recipes or larger targets.
- Evaluation is restricted to multiple-choice cloze-style tasks suited to small scales; math benchmarks remain unpredictable with the proxy metrics.
Experimental setup
Datasets: Dolma 1.7 Dolma 1.6++ C4 FineWeb-Pro FineWeb-Edu Falcon (RefinedWeb) Falcon+CC DCLM-Baseline
Benchmarks: MMLU ARC Challenge ARC Easy HellaSwag PIQA CommonsenseQA SocialIQA OpenBookQA BoolQ WinoGrande MBPP HumanEval
Metrics: decision accuracy prediction error percent of target compute budget
Baselines: Single-scale ranking 3-parameter scaling law (L(C) + Acc(L)) 2-parameter scaling law 5-parameter (N,D) scaling law Accuracy metric at target scale
Builds on: OLMo model ladder OLMES Pythia suite Chinchilla scaling laws DCLM
Closest papers by meaning
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.89)
- Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale (ICLR 2025 · similarity 0.88)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.88)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.