Research map / Learning theory and algorithms / Kernels and regression theory
Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses dynamical mean field theory on a Gaussian approximation of two-layer networks trained by gradient flow to show a timescale separation between feature learning, overfitting, and feature unlearning in the large-width limit.
Problem
Understanding how the training dynamics of overparametrized two-layer networks determine implicit bias, generalization, and when overfitting begins relative to feature learning.
Approach
Characterize gradient flow on two-layer networks with DMFT, replacing the non-Gaussian fitting errors by Gaussian processes with matching covariance, and show that the second-layer complexity acts as a slow variable, decoupling feature learning (fast) from overfitting (slow, time order m).
Key results
- With mean-field initialization, DMFT predicts test and train error converge to the Bayes error tau^2/2 in the first O(1) time regime, with train-test gap negligible.
- Overfitting and feature unlearning occur on times t = Omega(m): second-layer weights grow to order sqrt(m) and the test error rises above the Bayes error while train error goes to zero.
- For pure noise, gradient flow interpolates only if complexity gamma exceeds a threshold gamma*_GF(alpha) that converges as m grows; the threshold lies in (0,1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mean-field initialization instead of lazy initialization | two-layer network on a single-index model · single-index model, large-width limit | test error | Helped |
| continue training into the overfitting regime until interpolation | two-layer network with latent structure · single-index model, large-width limit | test error | Hurt |
| increase training time into the late regime | two-layer network with latent structure · single-index model, large-width limit | test error | Hurt |
| increase network width at fixed overparametrization ratio n/md=α | two-layer network · single-index model | overfitting onset time | Helped |
| initialize with complexity below the interpolation threshold, allowing complexity to grow during training | gradient flow on a two-layer network · single-index model | training error | Helped |
| initialize with complexity above the interpolation threshold | gradient flow on a two-layer network · single-index model | test error | Hurt |
| stop training at the end of the first phase rather than continue into overfitting | two-layer network with low-complexity initialization · single-index model | test error | Helped |
Surprising findings
- Training until interpolation is generally suboptimal; the best generalization tradeoff occurs at the end of the first feature-learning phase, and test error rises afterward.
- Feature unlearning: the projection of first-layer weights onto the latent subspace decays to order 1/sqrt(m) during overfitting.
Limitations
- DMFT is applied to a Gaussian approximation of the network's fitting errors; rigorous justification is only for times of order one or logarithmic scales.
- Analysis restricted mainly to single-index models with k=1 and symmetric initialization; the t=Theta(m) scaling is not precisely determined.
Experimental setup
Datasets: synthetic single-index model data (Gaussian inputs, k=1) pure noise data
Metrics: train error test error generalization error second-layer weight l1 norm
Baselines: neural tangent kernel theory lazy initialization
Builds on: dynamical mean field theory mean field theory of two-layer networks neural tangent kernel
Closest papers by meaning
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
- Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective (ICML 2024 · similarity 0.88)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.87)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.