Research map / Learning theory and algorithms / Kernels and regression theory
How Feature Learning Can Improve Neural Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a solvable two-layer linear projected-SGD model with evolving NTK, uses DMFT to derive neural scaling laws, and shows feature learning improves the loss exponent only for hard tasks (beta<1), validated on MLPs on the circle and CNNs on MNIST/CIFAR-5M.
Problem
Kernel (lazy) models of neural scaling laws fail to match scaling of feature-learning networks; it is unclear when feature learning changes power-law exponents of loss versus training time, model size, and compute.
Approach
A two-layer linear network with projected SGD whose first-layer features evolve (a finite-width NTK that changes) while a fixed random projection bottlenecks the model; DMFT yields closed-form scaling exponents, showing feature learning improves time scaling from beta to 2beta/(1+beta) for hard tasks (beta<1) but not for easy tasks (beta>1).
Key results
- For hard tasks (beta<1), feature learning changes the time-scaling exponent from beta to 2beta/(1+beta), nearly doubling it for small beta.
- For easy tasks (beta>1), the loss exponent matches the lazy model (t^-beta), so feature learning does not improve scaling.
- Compute-optimal exponent for hard tasks improves from alpha*beta/(alpha+1) to 2 alpha beta/(alpha(1+beta)+2); MNIST-1M scaling is well captured by the predicted exponent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Allow feature evolution rather than keeping features static | Scaling on hard tasks (β < 1) · Hard tasks | loss scaling exponent with training time: \beta -> \frac{2\beta}{1+\beta} | Helped |
| Allow feature evolution rather than keeping features static | Compute-optimal scaling on hard tasks (β < 1) · Hard tasks | compute-optimal loss exponent r_C: \frac{\alpha \beta}{\alpha + 1} -> \frac{ 2 \alpha \beta}{\alpha (1+\beta) + 2} | Helped |
| Allow feature evolution rather than keeping features static | Scaling on easy tasks (β > 1) · Easy tasks | loss scaling exponent with training time: \beta -> \beta | No clear effect |
| Allow feature evolution rather than keeping features static | Compute-optimal scaling on easy tasks · Easy tasks (1 < β < 2−1/α) | compute-optimal loss exponent r_C: \frac{\alpha\beta}{ \alpha\beta+1} -> \frac{\alpha\beta}{ \alpha\beta+1} | No clear effect |
| Allow feature evolution rather than keeping features static | Compute-optimal scaling on super-easy tasks · Super-easy tasks (β > 2−1/α) | compute-optimal loss exponent r_C: 1 - \frac{1}{2\alpha} -> 1 - \frac{1}{2\alpha} | No clear effect |
| Increase feature-learning parameter γ₀ | Residual CNN trained on CIFAR-5M · CIFAR-5M | loss scaling exponent | Helped |
| Increase feature-learning parameter γ₀ | CNN trained on CIFAR-5M · CIFAR-5M | loss scaling exponent | Mixed |
| Increase feature-learning parameter γ | Deep transformer · Language modeling, next-token prediction | cross-entropy loss scaling exponent: roughly twice the exponent | Helped |
Surprising findings
- On CIFAR-5M, the residual CNN's loss exponent at large feature-learning rate first follows the prediction but later enters a regime with a larger exponent than the theory predicts.
- SGD noise does not significantly alter scaling behavior for hard tasks, consistent with recent empirical findings that SGD noise has little impact in online training.
Limitations
- Analysis uses MSE loss with SGD and no adaptive optimizers, unlike modern cross-entropy training.
- Learned features are restricted to linear combinations of initial features, which may not hold in finite-width networks.
- Theory disagrees with CIFAR-5M CNN scaling after sufficient training.
Experimental setup
Datasets: MNIST-1M CIFAR-5M synthetic functions on circle with power-law Fourier spectra
Benchmarks: MNIST-1M CIFAR-5M Sobolev-space function fitting on circle
Metrics: test loss scaling exponent compute-optimal exponent
Baselines: lazy / random feature model (gamma=0) kernel regression
Builds on: random feature model scaling laws (Maloney et al.) dynamical mean field theory for SGD linear neural network rich/lazy regimes mean field / muP parameterization Chinchilla scaling law
Closest papers by meaning
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.91)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.90)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.89)
- The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.89)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.88)
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.