Research map / Optimization / Implicit bias and generalization
Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a two-layer NN trained with one gradient step on Gaussian mixture data with finite-rank-plus-identity covariances has the same training and generalization errors as an equivalent Hermite polynomial model in the proportional asymptotic limit.
Problem
Characterizing training and generalization error of two-layer networks after one gradient step under structured (mixture, low-rank-plus-identity covariance) data, where prior analyses assume isotropic or spiked Gaussian inputs.
Approach
Conditional Gaussian equivalence conditioned on the mixture component and the alignment with the gradient spike, plus a Hermite polynomial approximation whose degree depends on the strength parameter beta = log(eta*||Sigma||)/log(n), giving an equivalent polynomial ("Hermite") model.
Key results
- Hermite model with degree l=5 matches neural network generalization error across k/m ratios, activations ReLU, tanh, sigmoid (simulations with n=m=1000, 20 Monte Carlo runs).
- Higher strength parameter beta (larger eta*||Sigma||) yields lower generalization error; increasing alpha (shifting scaling to data spread) increases generalization error.
- On cGAN-generated Fashion-MNIST binary tasks, the degree-5 Hermite model achieves nearly the same generalization error as the neural network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the nonlinear activation with the finite-degree Hermite activation | Two-layer NN after one gradient step · Gaussian mixture simulations | generalization error | No clear effect |
| Replace the nonlinear activation with the finite-degree Hermite activation | Two-layer NN after one gradient step · Gaussian mixture simulations | training error | No clear effect |
| Use the degree-5 Hermite model instead of the neural network | Two-layer NN trained on cGAN-generated Fashion-MNIST data · Fashion-MNIST Class 0 vs. Class 1 | generalization error | No clear effect |
| Use the degree-5 Hermite model instead of the neural network | Two-layer NN trained on cGAN-generated Fashion-MNIST data · Fashion-MNIST Class 5 vs. Class 9 | generalization error | No clear effect |
| Increase alpha, shifting scaling from data spread toward learning rate | Two-layer NN · Gaussian mixture simulations, beta = 3/4^- | generalization error | Hurt |
| Increase beta, increasing the joint scaling of learning rate and data spread | Two-layer NN · Gaussian mixture simulations, alpha = 1/2 | generalization error | Helped |
| Increase alignment between the spiked directions of mixture components | Two-layer NN · Gaussian mixture regression | generalization error | Helped |
| Increase the covariance matrix effective rank | Two-layer NN · Gaussian mixture regression | generalization error | Helped |
| Vary the mixture ratio | Two-layer NN with tanh activation · Gaussian mixture classification | generalization error | No clear effect |
| Use completely imbalanced mixture ratios | Two-layer NN with ReLU or Sigmoid activation · Gaussian mixture classification | generalization error | Helped |
Surprising findings
- Higher data spread is more beneficial for generalization than a higher learning rate for the first layer at fixed beta.
- Higher covariance rank d_c lowers generalization error, attributed to larger maximum eigenvalues from random sampling.
Limitations
- Analysis restricted to beta in [0,1] (one-step regime) and single-index targets.
- Assumes zero-mean mixture components and a finite-rank-plus-identity covariance structure; real-data validation uses cGAN-generated preprocessed samples rather than raw data.
Experimental setup
Datasets: Fashion-MNIST
Benchmarks: Fashion-MNIST binary classification (T-shirt vs Trouser, Sandal vs Ankle boot)
Metrics: generalization error training error
Baselines: random feature model linear noisy model
Builds on: Gaussian universality for random features Gaussian equivalence for one-step gradient feature learning universality for Gaussian mixture GLMs
Closest papers by meaning
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.92)
- A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.88)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.87)
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.87)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.