Research map / Label-efficient and robust learning / Domain adaptation and generalization
Explaining Data Mixing Scaling Laws
Rui Dai, SHURAN ZHENG
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use theoretically grounded Extended Linear Regression model instead of the Additive law | data mixing loss prediction · Pile 17 domains, 64 1B models (fitting accuracy) | MRE (%): 2.209 -> 1.533 | Improved |
| use Extended Linear Regression instead of Extended Quantization model (ablation) | data mixing loss prediction · Pile 17 domains, 64 1B models (fitting accuracy) | MRE (%): 2.064 -> 1.533 | Improved |
| use extrapolated mixture for 700M/16B target instead of static mixture from 200M/8B (ablation) | Extended Linear Regression mixture · 4-domain 700M/16B | test loss | Improved |
| use extrapolated mixture for 1B/30B target instead of static 122M/10B mixture (ablation) | Extended Linear Regression mixture · 7-domain SlimPajama 1B/30B | test loss | Improved |
| small-scale-fitted law vs Additive law fitted across all scales including target | Extended Linear Regression mixture · 7-domain SlimPajama 1B/30B | test loss | No clear effect |
| use mixture predicted by Extended Linear Regression vs uniform and empirical laws | optimal mixture prediction · 4-domain 200M, 7-domain 122M and 310M, Pile 17-domain | test perplexity or loss | Improved |
Closest papers by meaning
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.92)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.90)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.88)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
