Research map / Architectures and efficiency / RNNs and normalization
On the Origin of Neural Scaling Laws: from Random Graphs to Natural Language
Maissam Barkeshli, Alberto Alfarano, Andrey Gromov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace unbiased walk on ER graph (kappa=0) by power-law-biased transitions (kappa=1) (ablation) | 2-layer transformer on ER 8K/50K · ER 8K/50K | power-law fit quality and exponents: alpha_D 1.028 -> 0.925 | Improved |
| include the irreducible loss offset E_D in the L(N)_D fit (ablation) | Pythia scaling fit · Pythia models | scaling exponent and fit MSE: nearly a factor of two in exponent, almost an order of magnitude in MSE | Improved |
| use neural network or kernel regression instead of 2d Chinchilla formula | L(N,D) fit · Chinchilla data | validation MSE: almost 2x improvement in average validation MSE | Improved |
| use 1d fit L(D)_N instead of 2d Chinchilla fit | L(D)_N fit · Chinchilla data | MSE: 4.77 x 10^-4 -> 7.76 x 10^-5 | Improved |
| use muP instead of standard parameterization | 2-layer transformer on Fineweb-edu · Fineweb-edu | mean alpha_D: 0.354 -> 0.501 | Improved |
| include embedding parameters in N (ablation) | 2-layer transformer SP · Fineweb-edu | compute-optimal exponents: (a,b) = (0.742, 0.386) when excluded; a/b approx 0.94 when included | Mixed |
| increase data complexity from language bigram to T1L | 2-layer transformer · bigram vs T1L | mean alpha_D: 0.977 -> 0.550 | Hurt |
| increase data complexity from T1L to T2L | 2-layer transformer · T1L vs T2L | mean alpha_D: 0.550 -> 0.497 | Hurt |
| use 4-layer instead of 2-layer transformer on T-n-L data | transformer · T2L vs T4L | scaling results | No clear effect |
| use 4 layers instead of 2 layers | muP transformer on Fineweb-edu · Fineweb-edu | mean alpha_D: 0.501 -> 0.506 | No clear effect |
| replace power law fit by exponential fit (ablation) | 1d scaling fits · ER 8K/50K | MSE of L(N)_D fit: 1.07 x 10^-8 -> 7.03 x 10^-8 | Hurt |
| add power-law structure via Barabasi-Albert graph | ER vs BA random walk · random graphs | L(N)_D power-law fit quality | Hurt |
Closest papers by meaning
- Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.91)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.90)
- How Do Large Language Monkeys Get Their Power (Laws)? (ICML 2025 · similarity 0.90)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.90)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
