Research map / Optimization / Nonconvex and smooth optimization
A Stability Analysis of AdamW: Unstable Equilibria and Non-Convergence
Baiyu Su, Lizhang Chen, Qiang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical dynamical-systems analysis of AdamW showing its fixed points correspond to stationary points of a constrained, regularized problem, with only some being stable and non-convergence possible.
Problem
Understanding the behavior of AdamW, which is widely used in deep learning but poorly understood theoretically, particularly its convergence and the stability of its fixed points.
Approach
Treats AdamW as a dynamical system and shows it admits an implicit fixed-point objective whose fixed points are stationary points of a constrained and regularized optimization problem; stability of these fixed points depends on hyperparameters, and some fixed points are unstable, implying non-convergence.
Key results
- AdamW's fixed points coincide with stationary points of a constrained and regularized optimization problem.
- Not all fixed points of AdamW are stable; stability depends on hyperparameters.
- Some unstable fixed points imply AdamW does not converge to them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rescale AdamW weight decay by (|m|+e)/(sqrt(v)+e) (AdamMe) | AdamW · GPT-2-Small (124M) | validation loss: 3.355 -> 3.317 | Improved |
| use beta1 > beta2 with quadratic objective f(x)=(x-0.5)^2 | AdamW · 1D toy objective | stability of equilibrium: requires f''<9.1e-2 but f''=2 | Hurt |
| use beta1=0.9, beta2=0.99 with non-convex quartic | AdamW · 1D toy objective | convergence: f''(x*)=-1.20 vs required > -0.22 | Hurt |
| set beta1 = beta2 (equal momentum coefficients) (ablation) | AdamW · 1D stability analysis | stability condition | Improved |
| rescale weight decay (AdamMe) with b<=4a | AdamW ODE · continuous-time dynamics | convergence guarantee | Improved |
Surprising findings
- Some stationary points of the implicit objective are unstable under AdamW dynamics, so AdamW can fail to converge to them.
Limitations
- Full text unavailable; analysis details beyond the abstract could not be verified.
Experimental setup
Builds on: AdamW
Closest papers by meaning
- On the Convergence Direction of Gradient Descent (ICLR 2026 · similarity 0.89)
- Classification at the Edge of Stability: Unifying Self-Stabilization and Convergence Rates (NeurIPS 2026 · similarity 0.89)
- Beyond the Edge of Stability via Two-step Gradient Updates (ICML 2023 · similarity 0.89)
- Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization (ICML 2026 · similarity 0.89)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.88)
- On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
