Research map / Optimization / Optimizers
Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Adam can be viewed as discounted FTRL applied within the online-learning-of-updates framework, and analyzes its dynamic regret to explain the roles of momentum and discounting factors.
Problem
Existing theory of Adam mostly shows convergence rates no better than SGD and does not explain the roles of momentum and discounting (EMA) factors in Adam's update.
Approach
Cast optimizer design as online linear optimization of updates (online-to-nonconvex conversion); show Adam equals a discounted version of scale-free FTRL, and that discounting plus momentum give better dynamic regret, which translates into better optimization guarantees.
Key results
- Adam (discounted FTRL) with beta<1 achieves dynamic regret O(D G T^{2/3}) with beta=1-cT^{-2/3} on a lower-bound instance where non-momentum and undiscounted learners incur regret at least T-3 and (T-3)/2.
- Undiscounted (beta=1) scale-free FTRL has dynamic regret linear in path length P: O(GP sqrt(T)), versus O(sqrt(PT)) with discounting under tuning.
- On sparse hinge-loss classification, Adam with beta<1 outperforms SGD and beta=1 Adam variants in the experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add momentum (aggregate past gradients) instead of using only the latest gradient | Online learning of updates; optimizer counterparts include Adam versus SGD/AdaGrad · 2D online linear optimization, bounded domain [-1,1]^2 | dynamic regret: at least T-3 for non-momentum learners; O(DGT^{2/3}) for discounted Adam when P=O(D) | Helped |
| Add a discounting factor (β<1) instead of using β=1 | Discounted-FTRL / Adam · 2D online linear optimization, bounded domain [-1,1]^2 | dynamic regret: at least (T-3)/2 for β=1; O(DGT^{2/3}) for β<1 when P=O(D) | Helped |
| Use Adam with β<1 instead of SGD or Adam with β=1 | Sparse-data hinge-loss classification · Classification with z_i=e_i and z_i=c_i e_i, c_i∼Unif[0,2] | training loss | Helped |
| Use Adam with β=1 instead of β<1 | Sparse-data hinge-loss classification · Classification with sparse data and small learning rate | training loss | Hurt |
| Use SGD instead of Adam with β<1 | Sparse-data hinge-loss classification · Classification with sparse data and small learning rate | training loss | Hurt |
Surprising findings
- Undiscounted scale-free FTRL (beta=1) is shown to be a poor dynamic learner, with dynamic regret that grows linearly in the path length.
Limitations
- Analysis restricted to beta1 = beta2 and constant scaled learning rate.
- Dynamic regret bounds require knowledge of the environment (e.g., D or M) for tuning.
- Experiments limited to a small synthetic classification setup.
Experimental setup
Datasets: synthetic hinge-loss classification (d=100)
Benchmarks: synthetic sparse classification with regularized hinge loss
Metrics: training loss F(w) dynamic regret bounds
Baselines: SGD AdaGrad SGD with momentum scale-free FTRL (beta=1 Adam)
Builds on: online-to-nonconvex conversion scale-free FTRL AdaGrad online gradient descent
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps (NeurIPS 2024 · similarity 0.85)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.85)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.84)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.84)
- Discounted Adaptive Online Learning: Towards Better Regularization (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.