Research map / Architectures and efficiency / RNNs and normalization
Scaling Laws for Optimal Data Mixtures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes additive and joint scaling laws that predict the loss on a target domain as a function of model size, training tokens and data mixture weights, fit on small runs and used to find optimal mixtures for LLM, multimodal and vision pretraining.
Problem
Selecting data domain mixture weights for large-scale pretraining is done by costly trial and error, and no principled method predicts how mixture weights interact with model size and training tokens.
Approach
Extend Chinchilla-style power laws with a term depending on domain weights h, either additively (only the bias term E depends on h) or jointly (the A and B coefficients also depend on h), then minimize the fitted law over the simplex to get optimal mixtures that transfer to larger scales.
Key results
- Additive and joint laws extrapolate from small models to a 3B LLM and 8B NMM with low MRE, e.g. average MRE around 1.4% (additive) and 1.3% (joint) for LLM in the average-of-models table.
- The 7B LLM trained with the OpenHermes-optimized mixture scores highest on CORE (58 vs 52 base) and MMLU (37 vs 25 base).
- For the 7B LLM, the mixture optimized for average training loss improves CORE to 56 and reaches MMLU 32, compared with 53 for uniform weights.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the training mixture for OpenHermes loss instead of using uniform weights | 7B LLM · CORE tasks | median score: 53 -> 58 | Helped |
| Optimize the training mixture for OpenHermes loss instead of using the standard SlimPajama distribution | 7B LLM · MMLU | accuracy: 25 -> 37 | Helped |
| Use the additive scaling law instead of the simple scaling law (ablation) | LLM scaling-law prediction · LLM average | MRE (%): 1.70 -> 1.39 | Helped |
| Use the joint scaling law instead of the additive scaling law (ablation) | LLM scaling-law prediction · LLM average | MRE (%): 1.39 -> 1.30 | Helped |
| Allow scaling exponents to depend on domain weights, beyond the joint law (ablation) | LLM scaling-law prediction · LLM average | MRE (%): 1.30 -> 1.21 | Helped |
| Use the additive scaling law instead of the simple scaling law (ablation) | LVM scaling-law prediction · LVM average | MRE (%): 2.31 -> 2.55 | Hurt |
| Use the joint scaling law instead of the additive scaling law (ablation) | LVM scaling-law prediction · LVM average | MRE (%): 2.55 -> 2.21 | Helped |
| Use the joint rather than additive law for cosine-schedule extrapolation (ablation) | LLM scaling-law prediction · LLM, cosine schedule, 1B | average MRE (%): 0.76% -> 0.54% | Helped |
Surprising findings
- Optimal mixtures for a domain typically lie at the corner of the simplex, i.e. the optimal weight for a target domain is usually that domain itself, and optimal mixtures differ from the target mixture except at corners.
- Simpler additive law gave lower MRE than joint law in some cases at very few runs; the parameterization of alpha and beta as functions of h did not help.
Limitations
- Focuses on pretraining; continual pretraining and finetuning not studied.
- Assumes no data repetition (infinite stream per domain) and a fixed mixture throughout training.
- Predicts generic target loss rather than downstream task performance directly.
Experimental setup
Datasets: SlimPajama The Pile OpenHermes DCLM Obelics DFN COYO HQITP C4
Benchmarks: CORE MMLU ARC-easy ARC-challenge BoolQ PIQA SIQA HellaSwag WinoGrande OpenHermes loss
Metrics: mean relative error (MRE%) average loss downstream accuracy
Baselines: uniform mixture SlimPajama default proportions Ye et al. mixture laws (M1-M4) RegMix Ge et al. 2024 domain scaling law simple additive law full scaling law
Builds on: Chinchilla scaling laws Llama architecture AIMv2 RegMix
Closest papers by meaning
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.91)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.91)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.90)
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models (NeurIPS 2024 · similarity 0.90)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.