Research map / Optimization / Optimizers
On the Convergence of Adam and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · Synthetic online convex problem | average regret | Helped |
| Use Adam's exponential moving average normalization | Adam · Synthetic online convex problem | average regret | Hurt |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · Synthetic online convex problem | iterate convergence | Helped |
| Use Adam's exponential moving average normalization | Adam · Synthetic stochastic convex problem | iterate convergence | Hurt |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · MNIST logistic regression | train loss | Helped |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · MNIST logistic regression | test loss | Helped |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · MNIST logistic regression | robustness to parameter changes | Helped |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · CIFAR-10 CIFARNet | train loss | Helped |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · CIFAR-10 CIFARNet | accuracy | Helped |
| Replace Adam's current squared-gradient average with the running maximum of those averages | Adam · CIFAR-10 CIFARNet | test loss | Helped |
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.94)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.92)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.90)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.90)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.89)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.