Research map / Architectures and efficiency / RNNs and normalization
A Dynamical Model of Neural Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes a linear random-feature student trained by gradient flow/SGD on power-law features using dynamical mean field theory, deriving time, data, and model scaling bottlenecks and an asymmetric compute-optimal scaling rule (training time should grow faster than model size).
Problem
Explaining neural scaling laws, including why test loss exponents with training time and model size differ, the compute-optimal allocation between model size and training steps, and how finite-width and finite-data corrections accumulate over training.
Approach
Models a randomly projected linear student (features A psi(x)) trained on a power-law teacher; uses dynamical mean field theory to get exact time-dependent learning curves, showing a time bottleneck with exponent (a-1)/b distinct from the model/data exponent, which yields an asymmetric compute-optimal rule.
Key results
- Time bottleneck gives test loss ~ t^{-(a-1)/b}, while data and model bottlenecks give P^{-min(a-1,2b)} and N^{-min(a-1,2b)}, so time and model exponents differ.
- Compute-optimal loss scales as C^{-(a-1)/(1+b)} with training time growing faster than model size; the prediction matches the linearized ResNet on CIFAR-5M.
- For ResNets on CIFAR-5M, the spectral decay and task-power fits (lambda_k ~ k^{-2.0}, k^{-0.15}) give a predicted compute-optimal exponent of about 0.05, while feature-learning networks scale differently.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase training time t with power-law features | Random feature model (DMFT) · Power-law features, time bottleneck | test loss: t^{-(a-1)/b} | Helped |
| increase dataset size P with power-law features | Random feature model (DMFT) · Power-law features, data bottleneck | test loss: P^{-\min\{a-1,2b\}} | Helped |
| increase model size N with power-law features | Random feature model (DMFT) · Power-law features, model bottleneck | test loss: N^{-\min\{a-1,2b\}} | Helped |
| allocate compute to training time faster than to model size | Random feature model (DMFT) · Power-law features, compute-optimal regime where m = a-1 | test loss: t \sim C^{\frac{b m}{a-1+b m}} , N \sim C^{\frac{a-1}{a-1+b m}}; \mathcal{L}^\star(C) \sim C^{-\frac{a-1}{1+b}} | Helped |
| use feature learning rather than the linearized kernel regime | Networks on CIFAR-5M · CIFAR-5M | compute-optimal test loss scaling | Helped |
| increase model size rather than ensemble count at fixed compute | Random feature model · Fixed compute C = N E t | test loss | Helped |
Surprising findings
- Feature-learning networks outside the kernel regime obtain substantially better compute-optimal scaling than the linearized random-feature theory predicts.
Limitations
- Feature learning is not captured; kernel evolution is needed to explain the accelerated scaling of feature-learning networks.
- Results rely on proportional-limit DMFT and Gaussian-equivalence assumptions on the projection matrix A and data.
Experimental setup
Datasets: CIFAR-5M wikitext
Benchmarks: CIFAR-5M wikitext
Metrics: test loss train loss compute-optimal loss scaling exponent
Baselines: Chinchilla compute-optimal scaling rule
Builds on: neural tangent kernel dynamical mean field theory random feature models Chinchilla compute-optimal scaling
Closest papers by meaning
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.92)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.92)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.92)
- Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (ICML 2024 · similarity 0.91)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.91)
- How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.