Research map / Optimization / Optimizers
A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves last-iterate almost-sure and L1 convergence of Adam, plus non-asymptotic sample complexity, under L-smoothness and the ABC inequality (weaker than bounded-gradient assumptions used before).
Problem
Existing convergence analyses of Adam rely on stringent assumptions (almost surely bounded stochastic gradients, uniformly bounded gradients, or reshuffling), stronger than those used for SGD, and often lack almost-sure or L1 convergence results.
Approach
A framework based on an auxiliary momentum-decoupled variable u_t, a Lyapunov-type approximate descent inequality with a product scaling factor that absorbs the adaptive learning-rate change term, and martingale tools (Burkholder inequality, first-hitting-time decomposition) that works under the ABC inequality rather than bounded-gradient assumptions.
Key results
- Under L-smoothness and ABC inequality, Adam's gradient norm converges to zero almost surely and in L1 (last iterate) for gamma>1, delta>0.
- Non-asymptotic bound on average squared gradient norm of O(ln T / sqrt(T)) for gamma>1, delta=0, matching SGD-like rates up to log factor.
- Framework removes the almost sure bounded-gradient assumption required by prior Adam analyses to get L1 convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| analyze Adam under the ABC inequality and L-smoothness instead of requiring almost surely bounded stochastic gradients | Adam · Smooth non-convex optimization | convergence guarantees | Helped |
Limitations
- Hyperparameter class restricted (beta_2,t = 1 - 1/t^gamma, eta_t = t^-(1/2+delta)) and independent of T, causing extra ln T factor.
- Purely theoretical; no experiments reported.
- Asymptotic results rely on gamma>1 and delta>0 and the complexity bound is not fully optimal.
Experimental setup
Metrics: sample complexity bound almost sure convergence L1 convergence of expected gradient norm
Baselines: Wang et al. coordinate affine noise variance analysis Zhang et al. almost surely bounded gradient analysis Défossez et al. Li et al. reshuffling analysis
Builds on: Adam SGD ABC inequality Burkholder inequality
Closest papers by meaning
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.91)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.91)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.91)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.91)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.90)
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.