Research map / Optimization / Implicit bias and generalization
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reasonable fixed step size that reaches the Edge of Stability instead of a stable step size | Full-batch gradient descent on VGG-BN · CIFAR-10, VGG-BN, 99% accuracy | iterations: 16,622 -> 329 | Helped |
| Use a larger step size that enters the Edge of Stability rather than a stable step size | Full-batch gradient descent · CIFAR-10, cross-entropy | iterations to train: 3700 -> 1000 | Helped |
| Drop the learning rate after reaching the Edge of Stability | Full-batch gradient descent · CIFAR-10, 5k subset | sharpness: 2/200 -> 2/300 | Helped |
| Use cross-entropy instead of MSE loss (ablation) | Neural-network gradient descent · Classification tasks | sharpness | Mixed |
| Increase network width (ablation) | Fully-connected tanh networks with NTK parameterization · CIFAR-10, 5k subset | maximum sharpness | Hurt |
| Increase network width (ablation) | Neural networks trained with gradient flow · CIFAR-10 | progressive sharpening | Hurt |
| Increase network depth (ablation) | Neural networks trained with gradient flow · CIFAR-10 | progressive sharpening | Helped |
| Use a larger dataset (ablation) | Neural networks trained with gradient flow · CIFAR-10 subsets | progressive sharpening | Helped |
| Use an easier task or smaller dataset (ablation) | Neural networks trained with gradient flow · CIFAR-10 and other tasks | sharpness increase | Hurt |
| Use a shallow network (ablation) | Neural networks trained with gradient flow · CIFAR-10 | sharpness increase | Hurt |
| Add batch normalization while using MSE loss (ablation) | CNN with ReLU activations · CIFAR-10 | sharpness | Hurt |
| Use ReLU or hardtanh activations (ablation) | Neural-network gradient descent · Training objectives with non-differentiable components | sharpness at onset of instability: a bit less than 2/η | Mixed |
Closest papers by meaning
- Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability (ICLR 2023 · similarity 0.94)
- Understanding Edge-of-Stability Training Dynamics with a Minimalist Example (ICLR 2023 · similarity 0.93)
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares (NeurIPS 2025 · similarity 0.92)
- A Minimalist Example of Edge-of-Stability and Progressive Sharpening (NeurIPS 2025 · similarity 0.89)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.89)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.