Research map / Architectures and efficiency / RNNs and normalization
Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that compute-optimally trained models' normalized loss curves collapse onto a universal curve (supercollapse with LR decay), explained via power-law scaling and a quadratic SGD-noise model of LR schedules.
Problem
Understanding how neural network training dynamics scale when model size and training time grow together under compute-optimal allocation, beyond final losses and optimal hyperparameters.
Approach
Affinely normalize loss curves (subtracting estimated irreducible loss and dividing by seed-specific final reducible loss) across model sizes at compute-optimal horizons; they collapse to one universal curve, and LR decay reduces collapse deviation below the per-model seed noise floor. A quadratic SDE noise model with trace of preconditioned gradient covariance predicts loss curves across LR sched
Key results
- Normalized loss curves collapse across model sizes for transformers on CIFAR-5M and chess and for MLPs; with LR decay the collapse deviation falls below the per-model seed noise floor (supercollapse) for up to about half of training.
- A single shared alpha=0.21 in a simple trace-of-gradient-covariance model predicts loss curves across LR schedules, model sizes and horizons on CIFAR-5M.
- Replacing muP with constant learning rate, or using a suboptimal data exponent (1.2 vs 1.02), visibly breaks the collapse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learning-rate decay to the training schedule | Compute-optimally trained models · CIFAR-5M, Lichess, and MLP regression | collapse deviation | Helped |
| Replace μP learning-rate scaling with a constant learning rate across widths (ablation) | MLP regression · MLP regression | collapse quality | Hurt |
| Increase the data exponent γ from 1.02 to 1.2 (ablation) | Transformer trained on chess · Lichess | collapse quality: 1.02 to 1.2 | Hurt |
| Set the loss offset away from the estimated irreducible loss L₀ (ablation) | Compute-optimally trained models · CIFAR-5M | collapse quality | Hurt |
| Normalize by the expected final loss instead of the stochastic final loss for each realization (ablation) | Normalized loss curves · Learning-rate decay phase | collapse deviation | Hurt |
Surprising findings
- Normalizing by the stochastic final loss of each seed, rather than the expected final loss, is what yields supercollapse, with collapse deviation dropping below seed noise.
Limitations
- Theory relies on multiple approximations (quadratic loss, small learning rate, slowly varying Hessian) and power-law assumptions taken as given.
- Cross-entropy case with irreducible loss is not explained theoretically.
- Experiments only at small scale (about 10M to 80M parameters) as a proof of concept.
Experimental setup
Datasets: CIFAR-5M Lichess standard chess games synthetic power-law Fourier regression
Metrics: collapse deviation normalized loss curve variance test loss
Baselines: constant learning rate schedule standard parameterization
Builds on: neural scaling laws muP Chinchilla compute-optimal scaling noisy quadratic model
Closest papers by meaning
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.89)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.88)
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.88)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.