Research map / Optimization / Implicit bias and generalization
Understanding the unstable convergence of gradient descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a step size greater than 2/L on a convex quadratic | Gradient descent · Convex quadratic cost | convergence | Hurt |
| Flatten the quadratic cost with tanh | Gradient descent with η = 2/39 · Flattened quadratic cost | convergence | Helped |
| Use tanh instead of a linear activation | Single hidden neuron network · Single-neuron fitting task | convergence | Helped |
| Add activation functions instead of using a linear network | Two-hidden-layer fully connected network, width 200 · CIFAR-10, 5,000 examples | convergence | Helped |
| Increase the step size into the unstable regime | Full-batch GD with tanh activations · CIFAR-10 | relative progress ratio | Mixed |
| Use a larger step size in the unstable regime | Full-batch GD · CIFAR-10 | training speed | Helped |
| Use unstable-regime SGD with minibatch size 32 and η = 2/100 | ReLU network · CIFAR-10 | expected relative progress ratio | Mixed |
Closest papers by meaning
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.91)
- On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent (ICML 2020 · similarity 0.91)
- Global Convergence and Stability of Stochastic Gradient Descent (NeurIPS 2022 · similarity 0.90)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.89)
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares (NeurIPS 2025 · similarity 0.89)
- Derivatives of Stochastic Gradient Descent in parametric optimization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.