Research map / Architectures and efficiency / RNNs and normalization
Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Data Optimization (ADO), an online method that adjusts pretraining domain mixtures during training using per-domain power-law scaling laws fit on the fly, with no proxy models.
Problem
Choosing the pretraining data mixture across heterogeneous domains under limited compute; existing methods need proxy models or repeated small-scale experiments, adding cost and complexity.
Approach
Fit a power-law scaling law to each domain's training loss within a single run, compute each domain's learning potential (learning speed times reducible loss), weight it by a credit-assignment score based on recent sampling, and update the sampling mixture online with temporal averaging.
Key results
- At 124M scale, ADO achieves the highest average zero-shot accuracy (0.470) vs. 0.463 for Natural and 0.454 for the Pile default weights.
- At 1.3B scale, ADO achieves average zero-shot accuracy 0.590 vs. 0.585 for Natural and 0.575 for DoReMi, and wins on 4 of 7 tasks.
- Scaling-law fitting adds under 20 minutes to a 3.5-day 1.3B training run on TPU v3-128 (~0.4% overhead).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADO data-mixture optimization instead of the strongest baseline | 124M language model · Seven downstream tasks, average | accuracy: 0.461 -> 0.470 | Helped |
| Use ADO data-mixture optimization instead of the strongest baseline | 1.3B language model · Seven downstream tasks, average | accuracy: 0.585 -> 0.590 | Helped |
| Use ADO instead of Natural data mixture | 124M language model · LAMBADA | accuracy: 0.401 -> 0.371 | Hurt |
| Use ADO instead of Natural data mixture | 1.3B language model · Downstream tasks | accuracy | Mixed |
| Use ADO instead of Natural data mixture | 1.3B language model · Pile validation set | validation loss | Hurt |
| Use ADO instead of Natural data mixture | 1.3B language model · SlimPajama validation set | validation loss | Helped |
| Use ADO instead of Natural data mixture | 1.3B language model · FineWeb subset | validation loss | Helped |
Surprising findings
- A simple Natural (token-proportional) mixture is a very competitive baseline, second-best on average downstream accuracy at both scales.
- ADO slightly underperforms the Natural mixture on Pile validation loss, though it beats it on SlimPajama and FineWeb validation loss.
Limitations
- Only tested at 124M and 1.3B scales on the Pile; scaling to larger models is untested.
- Domain scaling laws ignore inter-domain interactions and learning rate schedules, and tend to overestimate final loss.
- Does not account for downstream-relevant domains such as code for reasoning unless a prior is set.
Experimental setup
Datasets: The Pile SlimPajama FineWeb
Benchmarks: HellaSwag WinoGrande PIQA ARC-E SciQ LogiQA2 LAMBADA Pile validation loss SlimPajama validation loss FineWeb validation loss
Metrics: zero-shot accuracy validation loss perplexity
Baselines: Pile default weights DoReMi ODM Natural mixture Balanced mixture
Builds on: DoReMi DoGE ODM Chinchilla-style neural scaling laws
Closest papers by meaning
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.88)
- Compute-Constrained Data Selection (ICLR 2025 · similarity 0.87)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.87)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.86)
- Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond (ICML 2024 · similarity 0.86)
- Adaptive Data Analysis for Growing Data (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.