Research map / Optimization / Implicit bias and generalization
Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes SGD on a two-layer quadratic-activation network learning an extensive-width additive model with power-law second-layer weights, deriving scaling laws for risk via a matrix Riccati ODE and monotonicity arguments.
Problem
Understanding optimization and sample complexity of gradient-based training of two-layer quadratic networks in the extensive-width regime (r ~ d^beta) with power-law decaying second-layer coefficients, and whether this yields neural scaling laws.
Approach
Reduces the population and online SGD (Stiefel) dynamics to a matrix Riccati ODE/difference equation and uses monotonicity with respect to initialization, plus comparison with decoupled diagonal bounding sequences, to obtain operator-norm control and sharp power-law risk curves for k=2 (quadratic) targets where prior decoupling arguments fail.
Key results
- For alpha>0.5, population risk scales as t^{-(2alpha-1)/alpha} + r_s^{-(2alpha-1)} under gradient flow and online SGD with step size eta ~ 1/(d polylog d).
- For isotropic alpha=0, online SGD sample complexity n ~ d r polylog(d), improving the poly(r) dependence of prior work.
- ReLU and He1+He2 experiments with population gradient descent show empirical power-law exponents roughly matching the predicted (1-2alpha)/alpha for the quadratic setting, though with deviations (e.g., -1.26 vs ideal -4/3 for alpha=3/2 in ReLU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online SGD on the Stiefel manifold with the stated step-size regime | Two-layer quadratic NN · Online SGD (Stiefel), light-tailed (α>0.5) | risk scaling: (\eta\bar{t})^{- \frac{2 \alpha - 1}{\alpha}} + r_s^{- (2 \alpha - 1)} | Helped |
| Use feature-learning SGD rather than lazy-regime training | Quadratic target learning · Light-tailed (α>0.5) | sample complexity: n \simeq d \, \mathrm{polylog}(d); n\gtrsim d^2 | Helped |
| Increase the decay exponent from α=0 to α>0 | Quadratic NN gradient flow · Population risk trajectory | risk scaling: \mathcal{R}(t r \log d) \to (1 - C t^{\frac{1 - 2 \alpha}{\alpha}})_+ \vee (1 - \varphi^{1 - 2\alpha})_+; \mathcal{R}(t | Mixed |
| Increase student width r_s | Quadratic NN gradient flow · Light-tailed (α>0.5) | risk: r_s^{- (2 \alpha - 1)} | Helped |
| Train the directional component with online SGD and then fine-tune the radial component | Online SGD (Stiefel) · Discrete dynamics | risk trajectory: o_d(1) | Helped |
Surprising findings
- The Riccati flow is monotone in its initialization but the alignment trajectory itself is not monotone in time.
- Empirical power-law exponents for ReLU networks deviate notably from the theoretical prediction at alpha=3/2 (e.g., -1.26 vs -4/3), and the paper conjectures multi-phase behavior from higher Hermite modes.
Limitations
- For intermediate alpha in (0,0.5), the sample complexity bound is suboptimal by a factor r^alpha (conjectured optimal is dr).
- Requires orthonormal teacher directions and specific initialization/high-probability regimes; main analysis uses Stiefel SGD rather than standard Euclidean SGD.
- Fine-tuning step is needed for the radial component; standard Euclidean discretization is only conjectured to work.
Experimental setup
Datasets: synthetic Gaussian data from teacher quadratic network
Metrics: population risk (MSE) alignment scaling exponent
Baselines: Standard scaling-law power-law hypothesis (neural scaling laws) Prior quadratic NN analyses with poly(r) sample complexity
Builds on: Hermite polynomial information exponent analysis matrix Riccati ODE online SGD on Stiefel manifold
Closest papers by meaning
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.91)
- High-dimensional Limit of SGD for Diagonal Linear Networks (NeurIPS 2026 · similarity 0.89)
- The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks (NeurIPS 2025 · similarity 0.89)
- Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit (NeurIPS 2024 · similarity 0.89)
- SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem (ICLR 2024 · similarity 0.88)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.