Research map / Optimization / Implicit bias and generalization
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase student hidden units K beyond teacher M, training only first layer (soft committee machine) | sigmoidal student · teacher-student online SGD, two-layer network | asymptotic generalisation error | Hurt |
| increase student hidden units K beyond teacher M, training only first layer | linear student · teacher-student online SGD, two-layer network | asymptotic generalisation error | Hurt |
| increase student hidden units K beyond teacher M, training only first layer | ReLU student · teacher-student online SGD, two-layer network | asymptotic generalisation error | Hurt |
| also train the second layer, then increase K | sigmoidal student · teacher-student online SGD, two-layer network | asymptotic generalisation error | Helped |
| also train the second layer, then increase K | linear and ReLU student · teacher-student online SGD, two-layer network | asymptotic generalisation error | No clear effect |
| add weight decay to SGD | student with L>0 · teacher-student online SGD, two-layer network | generalisation error | No clear effect |
| scale learning rate as 1/K for larger students | sigmoidal SCM · teacher-student online SGD, two-layer network | generalisation error | Hurt |
| normalise second-layer weights to 1/K instead of 1 (normalised SCM) | SCM · teacher-student online SGD, two-layer network | asymptotic generalisation error | Helped |
| use MNIST images as inputs instead of Gaussian inputs | SCM · teacher-student online SGD, two-layer network | eps_g vs L curve | No clear effect |
| reduce training set size (finite data) | non-linear student · teacher-student online SGD, two-layer network | asymptotic generalisation error | Mixed |
Closest papers by meaning
- How Gradient descent balances features: A dynamical analysis for two-layer neural networks (ICLR 2025 · similarity 0.92)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.90)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
- The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents (ICML 2024 · similarity 0.90)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.89)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.