Research map / Optimization / Nonconvex and smooth optimization
Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a convergence analysis of Adam (and blockwise Adam) under coordinate-wise l_infinity smoothness, measured by the (1,1)-norm of the Hessian, showing that Adam's advantage over SGD relates to this non-rotation-invariant geometry.
Problem
Existing convergence analyses of Adam and SGD assume l2 smoothness, are rotation-invariant, and give the same rate, so they cannot explain why Adam outperforms SGD when training language models.
Approach
Replace spectral (l2) smoothness with coordinate-wise l_infinity smoothness, where the relevant complexity is the (1,1)-norm of the Hessian (normalized by dimension). This yields an Adam (RMSProp) rate that depends on sum of coordinate smoothness constants instead of d times the spectral norm, and extends to blockwise Adam via a Phi-smoothness measure. Experiments on rotated losses, GPT-2 and ResN
Key results
- On the quadratic task, Adam's final loss worsens monotonically as the (1,1)-norm/d rises (0.00002 unrotated to 0.00254 for R4 with beta1=0.9), tracking the Hessian norm.
- On GPT-2 small, estimated smoothness is 2.3538 for Adam versus 4.2446 for AdaSGD and 14.3745 for rotated Adam; Adam converges faster than AdaSGD and rotated Adam.
- On ResNet18 with batch size 256, Adam reaches training loss 0.0359 versus 0.0723 for SGD, 0.0787 for AdaSGD and 0.0485 for rotated Adam, and its Hessian (1,1)-norm estimate is 0.0036 versus 1.5355 for AdaSGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rotate the quadratic loss with R₁ (ablation) | Adam · Quadratic loss, 100 steps | final loss: 0.00002 ± 0.00001 -> 0.00066 ± 0.00008 | Hurt |
| rotate the quadratic loss with R₂ (ablation) | Adam · Quadratic loss, 100 steps | final loss: 0.00002 ± 0.00001 -> 0.00134 ± 0.00007 | Hurt |
| rotate the quadratic loss with R₃ (ablation) | Adam · Quadratic loss, 100 steps | final loss: 0.00002 ± 0.00001 -> 0.00183 ± 0.00009 | Hurt |
| rotate the quadratic loss with R₄ (ablation) | Adam · Quadratic loss, 100 steps | final loss: 0.00002 ± 0.00001 -> 0.00254 ± 0.00008 | Hurt |
| use Adam instead of AdaSGD | optimizer · Quadratic loss, 100 steps | final loss: 0.00405 ±0.00021 -> 0.00002 ± 0.00001 | Helped |
| rotate the GPT-2 loss (ablation) | Adam · GPT-2 on OpenWebText | estimated Hessian smoothness: 2.3538 -> 14.3745 | Hurt |
| use Adam instead of AdaSGD | ResNet18 · CIFAR-10, batch size 16 | training loss: 0.114 -> 0.064 | Helped |
| rotate the loss (ablation) | Adam · ResNet18 on CIFAR-10 | training loss: 0.064 -> 0.0905; 0.0472 -> 0.0574; 0.0359 -> 0.0485; 0.0735 -> 0.0817 | Hurt |
| use a smaller peak learning rate, 3 × 10⁻⁴ instead of 6 × 10⁻⁴ (ablation) | adaptive optimizers on GPT-2 · GPT-2 on OpenWebText | training loss | Hurt |
| use a larger peak learning rate, 1.8 × 10⁻³ instead of 6 × 10⁻⁴ (ablation) | Adam · GPT-2 on OpenWebText | training loss | Helped |
| use a larger peak learning rate, 1.8 × 10⁻³ instead of 6 × 10⁻⁴ (ablation) | Rotated Adam · GPT-2 on OpenWebText | training loss | Hurt |
| use a larger peak learning rate, 1.8 × 10⁻³ instead of 6 × 10⁻⁴ (ablation) | AdaSGD · GPT-2 on OpenWebText | training success | Hurt |
Surprising findings
- Rotated Adam performs even worse than AdaSGD, which is rotation-equivariant, on GPT-2, suggesting rotation-invariant assumptions cannot explain Adam's advantage.
- AdaSGD training failed entirely at the larger GPT-2 learning rate of 1.8e-3, while Adam improved.
Limitations
- The (1,1)-norm smoothness measure is only an upper-bound surrogate; the analysis is for beta1=0 (RMSProp) and does not explain momentum's benefit.
- Experiments are limited to GPT-2 small and ResNet18, and rotations are structured random orthogonal matrices rather than Haar-random ones.
Experimental setup
Datasets: OpenWebText CIFAR-10 synthetic quadratic loss
Benchmarks: OpenWebText training and evaluation loss CIFAR-10 training loss and accuracy synthetic quadratic loss
Metrics: final training loss (1,1)-norm of Hessian over d Phi-smoothness estimate evaluation loss accuracy
Baselines: SGD AdaSGD rotated Adam SignGD
Builds on: Adam SignGD RMSProp AdaGrad analysis (Defossez et al.) AdaSGD
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.90)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.90)
- Understanding Adam Requires Better Rotation Dependent Assumptions (NeurIPS 2025 · similarity 0.88)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.88)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.