Research map / Optimization / Implicit bias and generalization
Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use averaged SGD with regularization tuned to target-function and NTK complexities | Overparameterized two-layer neural networks in the NTK regime · Regression under assumptions (A1)-(A4) | squared L2 generalization error rate: O(T^{\frac{-2r\beta}{2r \beta +1}}) | Helped |
| Use a sufficiently wide network to approximate the NTK dynamics | Averaged SGD for overparameterized two-layer neural networks · Regression in the NTK regime | squared L2 generalization error | Helped |
| Avoid requiring positivity of the NTK Gram matrix | Global convergence analysis of averaged SGD · Population-risk regression | squared L2 generalization error | Helped |
| Train the layer whose kernel specifies the target function (ablation) | Two-layer swish network · Synthetic regression with target specified by the output- or input-layer kernel | test error | Helped |
| Train a layer whose kernel does not specify the target function (ablation) | Two-layer swish network · Synthetic regression with misspecified target layer | test error | Hurt |
| Train both layers rather than only a target-matched layer (ablation) | Two-layer swish network · Synthetic regression with Bayes rules specified by either layer kernel | test error | No clear effect |
| Use a smooth approximation of ReLU under the uniform-sphere setting | Averaged SGD for two-layer neural networks · Regression with target specified by the ReLU NTK | squared L2 generalization error rate: O\left(T^{\frac{-2rd}{2rd + d - 1}}\right) | Helped |
Closest papers by meaning
- Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach Space (ICML 2023 · similarity 0.90)
- Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS 2019 · similarity 0.89)
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks (NeurIPS 2020 · similarity 0.89)
- A generalized neural tangent kernel for surrogate gradient learning (NeurIPS 2024 · similarity 0.88)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.88)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.