Research map / Optimization / Implicit bias and generalization
On the Unstable Convergence Regime of Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis proving that unstable convergence of gradient descent is an inherent property for twice-differentiable functions near strict local minima, via forward-invariance and an open set of stable initializations.
Problem
Explaining why gradient descent still converges in the long run despite loss oscillations when the learning rate exceeds 2/L or L-smoothness fails (the edge-of-stability / unstable convergence regime).
Approach
Establishes forward-invariance of GD in an open neighborhood of a strict local minimum for C2 functions without convexity or L-smoothness, and shows the set of stably converging initializations is open, so trajectories from outside it jump in after finitely many oscillating steps.
Key results
- Proves forward-invariance of GD in an open set U around a strict local minimum for C2 functions (Theorem 2), with a descent property f(θ-η∇f(θ)) - f(θ) ≤ -Cη||∇f(θ)||².
- Proves the set of stable initializations Uη is open (Theorem 3), so unstable convergence occurs only for initializations outside Uη and the loss decreases monotonically after entering it.
- Numerical 2D examples (e.g., f1 with λmax=2, max LR 1) show the unstable initialization set grows as learning rate increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize outside the stable-convergence set Uη while keeping the GD setup | Gradient descent (GD) · General twice-differentiable functions with a strict local minimum | loss trajectory | Mixed |
| Increase the learning rate | Gradient descent (GD) · f1 and f2 (Equations 43 and 44) | stable versus unstable initialization regions | Mixed |
Surprising findings
- Unstable initialization sets can be irregular and even disconnected in the numerical examples.
Limitations
- Assumes a strict local minimum with compact set of global minimums; cannot handle non-compact minimum sets such as (1-xy)^2.
- Only qualitative analysis; no convergence speed guarantees.
- Analysis restricted to full-batch GD, not SGD.
Experimental setup
Metrics: loss trajectory convergence set membership of initializations
Builds on: Ahn et al. 2022 unstable convergence analysis edge of stability (Cohen et al. 2021) descent lemma
Closest papers by meaning
- On the Convergence Direction of Gradient Descent (ICLR 2026 · similarity 0.92)
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares (NeurIPS 2025 · similarity 0.91)
- Beyond the Edge of Stability via Two-step Gradient Updates (ICML 2023 · similarity 0.91)
- Understanding the unstable convergence of gradient descent (ICML 2022 · similarity 0.91)
- Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond (ICML 2023 · similarity 0.91)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.