Research map / Optimization / Nonconvex and smooth optimization
Adam with model exponential moving average is effective for nonconvex optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis showing that a clipped Adam variant combined with model EMA of iterates achieves optimal (λ,ε)-stationarity rates for nonconvex nonsmooth optimization, and that coordinate-wise adaptivity gives a factor-d advantage when gradient scales vary across coordinates.
Problem
Existing theory of Adam fails to explain its practical components (momentum, discounting, EMA of iterates); prior analyses give rates that worsen with momentum or are no better than SGD, and output a random iterate rather than EMA in nonconvex settings.
Approach
Combine a discounted-to-nonconvex online-to-nonconvex conversion with scale-free FTRL run on discounted gradients (beta-FTRL), which yields clipped Adam (beta1=beta, beta2=beta^2) and outputs the model EMA; coordinate-wise application exploits per-coordinate Lipschitz constants.
Key results
- Clipped Adam with model EMA finds a (λ,(1+(G+σ)/C)ε)-stationary point with iteration complexity O((G+σ)^2 Δ λ^{1/2} ε^{-7/2}), matching the known lower bound up to conditions on λ.
- For smooth functions the method achieves the optimal O(ε^{-4}) complexity; for second-order smooth functions it achieves O(ε^{-7/2}).
- Under coordinate-wise heterogeneity, the coordinate-wise variant improves the iteration complexity by a factor of d over a non-adaptive counterpart with the same L2 structure.
Surprising findings
- Random-iterate output used in standard nonconvex analyses is replaced by model EMA, which wastes no computation.
Limitations
- Analyzed algorithm uses beta1=beta and beta2=beta^2 and omits bias correction, so it does not exactly match practical Adam.
- No experiments; guarantees are theoretical.
- When G and sigma are unknown, the complexity carries an extra (G+sigma)^{3/2} factor relative to the lower bound.
Experimental setup
Metrics: iteration complexity for (lambda,epsilon)-stationary point L1-stationarity iteration complexity
Baselines: SGD online gradient descent based discounted analyses signSGD-based momentum analysis
Builds on: online-to-nonconvex conversion Adam scale-free FTRL discounted-to-nonconvex conversion
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.87)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.87)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.87)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.