Research map / Optimization / Implicit bias and generalization
Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses statistical-mechanics ODEs for one-pass SGD in a two-layer student-teacher committee machine with power-law data covariance spectra, deriving generalization-error scaling in time, feature number, and plateau length for linear and erf activations.
Problem
Theoretical understanding of neural scaling laws (power-law generalization error vs. training time, data/feature count) is limited; prior analyses mostly used random feature or linear models with isotropic or simple data.
Approach
Extends the statistical-mechanics online-learning hierarchy of ODEs for two-layer networks trained with one-pass SGD to power-law covariance spectra with L distinct eigenvalues, closing the hierarchy via Cayley-Hamilton; derives power-law decay of error in time, plateau length dependence on L, and a transition from exponential to power-law convergence.
Key results
- For linear activation, generalization error decays as alpha^(-beta/(1+beta)) within a power-law time window, matching simulations.
- Plateau escape time scales roughly as M^2/(eta L) for large M and L in the erf committee machine.
- Asymptotic error with trainable features N_l scales as 1/N_l^beta when L^beta > N_l^beta; empirical CIFAR-5m spectrum gives beta approx 0.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of distinct covariance eigenvalues | Nonlinear student-teacher network with a power-law spectrum · Power-law spectrum | plateau length | Hurt |
| Increase the number of distinct covariance eigenvalues | Student-teacher network with a power-law spectrum · Power-law spectrum | asymptotic convergence speed | Hurt |
| Train more input features (increase N_l) | Linear student-teacher network · Power-law spectrum | asymptotic generalization error: ε_{g,\mathrm{asymp}} \sim \frac{1}{N_l^\beta} | Helped |
| Increase the number of distinct eigenvalues L | Nonlinear student-teacher network in the large-M, large-L regime · Power-law spectrum | plateau escape time: τ_{\mathrm{esc}} \sim \frac{M^2}{\eta L} | Helped |
| Use the exact solution instead of the small learning rate approximation | Linear student-teacher network · Power-law spectrum | generalization error | Hurt |
| Use a power-law covariance spectrum rather than isotropic input correlations | Student-teacher network · Power-law spectrum | convergence regime | Mixed |
Surprising findings
- Plateau length shortens and can disappear as the number of distinct eigenvalues L increases.
Limitations
- Analysis restricted to small learning rate (O(eta)) approximations and a heuristic plateau-length formula.
- Results for the plateau are derived for a simplified teacher-teacher configuration with equal diagonal and off-diagonal entries.
- Thermodynamic limit N to infinity with fixed ratio alpha; finite-size effects only checked numerically.
Experimental setup
Datasets: CIFAR-5m
Benchmarks: CIFAR-5m
Metrics: generalization error
Builds on: Saad and Solla online learning Gaussian equivalence property Bordelon et al. random feature scaling
Closest papers by meaning
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.92)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.91)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.90)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.89)
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.