Research map / Optimization / Implicit bias and generalization
On the Implicit Bias of Adam
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a second-order-in-step-size backward error analysis (modified ODE) for mini-batch and full-batch Adam and RMSProp, showing Adam's implicit bias penalizes the perturbed one-norm of the gradient for beta > rho but typically anti-penalizes it (rho > beta), supported by CIFAR-10 experiments.
Problem
Backward error analysis, which explains implicit regularization of gradient descent via modified loss functions, has not been applied to adaptive optimizers such as RMSProp and Adam; their implicit bias on the loss gradient is unknown.
Approach
Derive a piecewise ODE approximating the discrete Adam/RMSProp iterates up to O(h^2) local-to-global accuracy, and interpret its correction term as implicit regularization of a perturbed one-norm of the gradient (penalization if beta > rho, anti-penalization if rho > beta when epsilon is small), in contrast to GD's squared two-norm penalty.
Key results
- For full-batch Adam with small epsilon, the modified loss adds (h/2){(1+beta)/(1-beta) - (1+rho)/(1-rho)} times the one-norm of the gradient, so Adam anti-penalizes the gradient one-norm when rho > beta.
- Theorem gives local-to-global O(h^2) approximation of Adam iterates by a piecewise ODE; numerically the ODE trajectory stays close to Adam in infinity-norm over 100 epochs on MNIST.
- On CIFAR-10 with ResNet-50 full-batch Adam, increasing rho raises the perturbed gradient one-norm and lowers test accuracy, while increasing beta lowers the norm and raises test accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase ρ | Full-batch Adam with ResNet-50 · CIFAR-10 | test accuracy | Hurt |
| Increase ρ | Full-batch Adam with ResNet-50 · CIFAR-10 | perturbed gradient one-norm | Hurt |
| Increase β | Full-batch Adam with ResNet-50 · CIFAR-10 | test accuracy | Helped |
| Increase β | Full-batch Adam with ResNet-50 · CIFAR-10 | perturbed gradient one-norm | Helped |
Surprising findings
- Adam typically anti-penalizes the gradient one-norm (implicit anti-regularization), the opposite of the implicit regularization found for gradient descent.
- Larger rho relative to beta increases the perturbed one-norm and worsens generalization, suggesting adaptive methods are biased toward sharper regions.
Limitations
- Constant in the error bound blows up as epsilon goes to zero, so the approximation may fail for very small epsilon at later stages.
- Analysis covers full-batch case qualitatively; mini-batch first-order terms are left for future work.
- Assumes bounded partial derivatives up to order 4, which may fail near edge-of-stability.
Experimental setup
Datasets: CIFAR-10 MNIST
Benchmarks: CIFAR-10
Metrics: test accuracy perturbed one-norm of gradient infinity-norm distance between Adam iterates and ODE solutio
Baselines: gradient descent signGD SGD with momentum
Builds on: backward error analysis for gradient descent Adam RMSProp signGD
Closest papers by meaning
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.90)
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.89)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.88)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.