Research map / Optimization / Optimizers
Towards Understanding Adam Convergence on Highly Degenerate Polynomials
Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou, Zhi-Qin John Xu, Yaoyu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Adam with beta2 > critical value instead of GD or Momentum on degenerate loss | GD and Momentum · L(x)=x^k/k, k>=4 | convergence rate: sublinear power-law t^(-1/(k-2)) -> linear | Improved |
| increase degeneracy from k=2 to k>2 | Adam · L(x)=x^k/k | stability: k=2 loss spikes -> k>2 stable exponential convergence | Improved |
| require beta1 < beta2^(k/(2(k-2))) (k=4: beta1 < beta2) (ablation) | Adam · L(x)=x^4/4 phase diagram | convergence: beta2^1.0 < beta1 < beta2^0.75 gives spikes | Improved |
| raise beta1 above beta2^(0.75) (k=4) (ablation) | Adam · L(x)=x^4/4 phase diagram | final loss: beta1 > beta2^0.75 gives SignGD-like oscillation | Hurt |
| increase degeneracy order k above 4 | Adam · stability region | stable region size: k=4 stability guarantees k>4 | Improved |
| keep beta2 above (k-2)/k)^(2(k-2)) for RMSProp (ablation) | RMSProp · L(x)=x^4/4 | stability: beta2 > 0.0625 | Improved |
| replace ReLU with softmax in MLPs | GD vs Adam · MLP training | GD speed relative to Adam | Hurt |
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.91)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.89)
- Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients (NeurIPS 2026 · similarity 0.89)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.88)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
