Research map / Optimization / Nonconvex and smooth optimization
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Adam's exponentially averaged squared gradients with AdaGrad's averaging of past squared gradients while retaining first-order momentum | Adam-type methods · Non-convex stochastic optimization | convergence | Helped |
| Use AMSGrad's non-increasing quadratic normalization instead of Adam's exponential second-moment estimate | Adam · One-dimensional non-convex example with oscillating effective stepsizes | convergence | Helped |
| Use AMSGrad's adaptive normalization rather than SGD's fixed learning rate | SGD · Quadratic problem f(x) = 100 x^2 | convergence | Helped |
| Use a constant learning rate instead of a diminishing learning rate | Adam · One-dimensional non-convex example | convergence | Hurt |
| Use a constant learning rate of 0.01 or greater instead of a smaller learning rate | SGD · One-dimensional non-convex example | convergence | Hurt |
| Increase oscillation of effective stepsizes | Adam-type methods · Non-convex stochastic optimization | convergence | Hurt |
| Use AdaGrad's second-moment estimate with first-order momentum instead of AdaGrad without momentum | AdaGrad · MNIST and CIFAR-10 | training and testing performance | Helped |
| Use AdaGrad without momentum and its simpler adaptive weighting | Adam-type algorithms · MNIST | classification accuracy | Hurt |
| Use AdaFom rather than Adam or AMSGrad | Adam-type algorithms · CIFAR-10 | classification accuracy | Hurt |
| Use constant momentum rather than diminishing momentum | AMSGrad · Non-convex stochastic optimization | convergence | No clear effect |
| Add a positive ε to the denominator for numerical stability | AMSGrad and AdaFom · Non-convex stochastic optimization | convergence guarantee | No clear effect |
| Use a constant rather than diminishing stepsize, such as α_t = 1/√T | AMSGrad · Non-convex stochastic optimization | convergence rate | Helped |
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.92)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.89)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.89)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.89)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.88)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.