Research map / Optimization / Implicit bias and generalization
Special Properties of Gradient Descent with Large Learning Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use large learning rate (0.01) instead of small (0.001) in SGD | ResNet-18 without BN, CIFAR10 · CIFAR10 | test performance | Helped |
| amplify stochastic noise via SGD with repeats (k=10) at small learning rate (ablation) | SGD with lr 0.001 · CIFAR10 | test performance | Helped |
| SGD with repeats (k=10, lr 0.001) instead of large lr 0.01 (ablation) | ResNet-18 SGD · CIFAR10 | test performance gap to large-lr SGD | Hurt |
| gradient descent on toy 1D function: large learning rate allows escape from local minimum and convergence to global minimum; small lr converges to local minimum | toy function C_l · theory / toy example | convergence to global minimum | Helped |
| increase stochastic noise magnitude at small learning rate (Uniform noise) (ablation) | SGD on toy function · toy 1D function | convergence to global minimum | No clear effect |
| gradient flow vs GD: continuous-time limit (infinitesimal step) does not escape local minimum (ablation) | gradient flow · theory | escape of local minimum | No clear effect |
Closest papers by meaning
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.93)
- SGD with Large Step Sizes Learns Sparse Features (ICML 2023 · similarity 0.93)
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance (ICML 2021 · similarity 0.93)
- Asymptotic theory of SGD with a general learning-rate (NeurIPS 2025 · similarity 0.92)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.91)
- Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.