Research map / Optimization / Optimizers
Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a slow SDE for adaptive gradient methods (Adam, RMSProp, Adam-mini, Adalayer, Shampoo) showing Adam semi-gradient descends tr(Diag(H)^{1/2}) near the minimizer manifold, yielding sparser solutions than SGD's tr(H) under label noise.
Problem
Adam's implicit bias near minima is poorly understood; prior slow-SDE analyses cover only SGD, and it is unclear what sharpness Adam reduces and how its solutions differ from SGD.
Approach
Generalizes the slow SDE framework (originally for SGD) to a general class of adaptive gradient methods via a preconditioner-flow projection and an adaptive projection operator, showing Adam takes semi-gradient steps on the sharpness tr(Diag(H)^{1/2}) under label noise, rather than tr(H) for SGD.
Key results
- Adam recovers the sparse ground truth at about n_train=420 versus gradual decline for SGD in the diagonal net setting (d=10000, kappa=50).
- Theorem: slow SDE for AGMs tracks Adam's trajectory over O(eta^-2) steps with weak approximation error O~(eta^0.25).
- In deep matrix factorization with label noise, Adam attains higher test loss than SGD despite similar training error, with lower tr(Diag(H)^{1/2}) but higher tr(H).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adam instead of SGD | Sparse linear regression with diagonal net · Sparse linear regression with diagonal net | training points needed for ground-truth recovery | Helped |
| Use AdamE with positive λ instead of SGD (ablation) | Sparse linear regression with diagonal net · Sparse linear regression with diagonal net | training points needed for ground-truth recovery | Helped |
| Use Adam instead of SGD | Deep matrix factorization with label noise · Deep matrix factorization with label noise, L=2 and L=5 | test loss | Hurt |
| Use Adam instead of SGD | Deep matrix factorization with label noise · Deep matrix factorization with label noise, L=2 and L=5 | tr(H) | Hurt |
Surprising findings
- Adam's implicit bias is detrimental in deep matrix factorization: it generalizes worse than SGD there.
- AdamE with a small positive lambda shows the same sudden sparse recovery as Adam, suggesting the preconditioning mechanism drives the effect.
Limitations
- Analysis restricted to the 2-scheme regime where 1-beta2 = Theta(eta^2)
- Does not cover weight decay as in AdamW
- Results assume iterates stay near a smooth minimizer manifold and label-noise covariance condition; tr(Diag(H)^{1/2}) interpretation relies on diagonal Hessian
Experimental setup
Datasets: synthetic sparse linear regression (diagonal net, d=10000, kappa=50) synthetic deep matrix factorization with label noise
Benchmarks: sparse linear regression with diagonal linear networks deep matrix factorization with label noise
Metrics: test loss tr(H) tr(Diag(H)^{1/2}) training MSE test MSE
Baselines: SGD RMSProp AdamE
Builds on: slow SDE for SGD Adam RMSProp implicit gradient regularization
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.91)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.90)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.90)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.89)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.89)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.