Research map / Optimization / Implicit bias and generalization
How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient descent with decreasing weight decay, after a one-step feature learning stage and a convex norm-adjustment stage, locally converges to the teacher network of a regularized 2-layer ReLU network and recovers teacher neuron directions, via dual certificate and test function analysis.
Problem
Whether feature learning by gradient descent can happen at the final (local convergence) stage of training, not only in early stages, for 2-layer ReLU networks learning a low-dimensional teacher network.
Approach
Analyzes local convergence of regularized GD (with decreasing weight decay) for 2-layer ReLU networks with second-layer weights of either sign, showing a descent direction exists in the local region and that student neurons align with teacher neuron directions, using dual certificate and test function arguments for the structure of approximate minima.
Key results
- Theorem: with width m >= m0 = Otilde(1)(1/eps0)^{O(r)} and decreasing weight decay, GD reaches loss <= eps within T = O(1/(eta eps^2)) time with high probability.
- As eps goes to 0, each student neuron aligns with a teacher neuron direction (angle 0) or has zero norm.
- Gradient lower bound ||grad||_F^2 >= Omega(zeta^4/lambda^2) holds in the regime Omega(lambda^2) <= zeta <= O(lambda^{9/5}), extending prior positive-second-layer results to signed second layer with ReLU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add standard weight decay and decrease it across training epochs | Two-layer ReLU network in the local-convergence regime · Gaussian teacher-student setup | square loss: L(\vtheta)\le{\varepsilon} | Helped |
| Continue training first-layer weights in the final stage instead of fixing them after early training | Two-layer neural network · Gaussian teacher-student setup | teacher-neuron alignment: angle(\vw_i,\vw_j^*)=0 | Helped |
| Use weight decay to balance layer norms and reduce cancellation between neurons | Two-layer neural network with positive and negative output weights · Local convergence | gradient descent progress | Helped |
| Use one gradient step followed by norm adjustment before local convergence | Three-stage training algorithm · Gaussian teacher-student setup | optimality gap: O_{*}({\varepsilon}_0) | Helped |
Limitations
- Stage 2 uses a convex norm-adjustment step rather than full joint training, so the analysis does not cover the full training dynamics from initialization.
- Requires non-degeneracy assumptions (separated teacher neurons, non-degenerate second-order matrix H) and Gaussian inputs.
- Relies on preprocessing and a small enough step-size depending on 1/m and d.
Experimental setup
Metrics: square loss optimality gap angle between student and teacher neurons
Builds on: early-stage feature learning analysis (one-step GD) dual certificate analysis for sparse/measure recovery mean-field / infinite-width analysis of 2-layer networks information exponent
Closest papers by meaning
- A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.90)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.89)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.87)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.87)
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.87)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.