Research map / Optimization / Implicit bias and generalization
4+3 Phases of Compute-Optimal Neural Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes one-pass SGD on a power-law random features model with a deterministic-equivalent Volterra equation to derive compute-optimal model-size scaling laws, identifying 4 phases (+3 subphases) in the data/target complexity plane.
Problem
Determining how to choose model-parameter count to minimize loss under a fixed compute (FLOP) budget with effectively infinite data, i.e., compute-optimal neural scaling laws, in a solvable setting.
Approach
Uses a three-parameter power-law random features model trained with one-pass SGD; derives a deterministic-equivalent Volterra equation for the expected loss curve, then decomposes it into forcing and SGD-noise kernel terms to obtain phase-dependent compute-optimal exponents and d*(C) ~ C^0.5 in a broad region.
Key results
- Power-law fit of empirical compute-optimal front gives exponent 0.648 vs theoretical 0.643 at alpha=0.7, beta=1.2 setting.
- Optimal parameter count scales as d* ~ C^0.508 (Approach 1) and C^0.525 (Approach 2) vs theory C^0.5 in Phase II-III boundary example.
- Identifies 4 phases (+3 subphases) in the (alpha, beta) plane; d* = C^0.5 holds across a large portion including Phases III, Ib, IVa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the distortion-dominated Phase II regime (2α > 1, 2β > 1, β < α) | PLRF trained with one-pass SGD · Phase II | compute-optimal tradeoff | Mixed |
| Increase model-parameter count instead of training through the feature-distortion loss region | PLRF compute-optimal training across phases · All phases | compute allocation | Helped |
| Use the SGD-noise-dominated, distortion-constrained regime (Phase III) | PLRF trained with one-pass SGD · Phase III | optimal model-parameter-count exponent: d^{\star}(\f) = \f^{1/2} | Helped |
| Use the Phase IV compute-optimal learning rate rather than the maximally stable learning rate | PLRF trained with one-pass SGD · Phase IV | compute-optimal loss | Helped |
| Use batch size B = 1 | PLRF compute-optimal training · All phases | compute-optimal loss | Helped |
Surprising findings
- In all phases it never pays to solve through the F_ac (distorted feature) part of the loss curve; increasing parameter count is always better.
- Empirical scaling exponents measured with 1 A100-GPU-day still change, warning that empirically derived scaling laws may not be asymptotically reliable.
Limitations
- Theory is for linear power-law random features with B=1 one-pass SGD; nonlinear models not analyzed.
- Asymptotic exponents require very large d, which is numerically hard to reach, especially in Phase IV.
- Theorem proven only for alpha > 1/4 and avoids critical lines; some regimes rely on numerical evidence.
Experimental setup
Datasets: synthetic power-law random features data
Metrics: scaling law exponent eta optimal parameter exponent xi loss vs flops
Baselines: Chinchilla IsoFLOP approach Bordelon et al. dynamical model Lin et al. linear regression scaling laws
Builds on: Chinchilla IsoFLOP scaling power-law random features dynamical model of neural scaling laws Defilippis et al. dimension-free deterministic equivalents
Closest papers by meaning
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.92)
- Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (ICML 2024 · similarity 0.89)
- Complexity Scaling Laws for Neural Models using Combinatorial Optimization (NeurIPS 2025 · similarity 0.89)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.89)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.88)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.