Research map / Optimization / Implicit bias and generalization
Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that AdamW (full-batch) converging under non-increasing learning rates with divergent sum must converge to a KKT point of the l_inf-norm-constrained problem with radius 1/weight-decay, via a link between normalized steepest descent with weight decay and Frank-Wolfe and a tight bound on Adam's average update size.
Problem
Understanding what solution AdamW converges to; AdamW's advantage over Adam with l2 regularization lacks theoretical explanation, and it was unclear whether AdamW optimizes any regularized objective.
Approach
Adam is viewed as a smoothed SignGD (normalized steepest descent w.r.t. l_inf); normalized steepest descent with decoupled weight decay equals Frank-Wolfe over the norm ball of radius 1/lambda, so AdamW converges to KKT points of min over the l_inf ball of radius 1/lambda, provided beta1 <= beta2, using a new upper bound on Adam's average update size.
Key results
- Theorem: if AdamW converges under non-increasing learning rates with divergent sum, the limit is a KKT point of min over ||x||_inf <= 1/lambda of L(x); for convex L it is a constrained minimizer.
- On PTB with beta1=beta2 (0.99, 0.999) or beta1=0.9, beta2=0.95, the l_inf norm of AdamW iterates stays below 1/lambda (1 and 1/2 for lambda=1 and 2) while Adam's keeps increasing.
- For the synthetic quadratic, steepest descent w.r.t. l_inf outperforms l2 steepest descent, with weight decay improving normalized steepest descent in both norms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdamW with β1=β2=0.99 or β1=β2=0.999 | AdamW on PTB · PTB | parameter ℓ∞ norm | Helped |
| Use β1=0.9 and β2=0.95 | AdamW on PTB · PTB | parameter ℓ∞ norm | Helped |
| Use β1=0.9 and β2=0.999, breaking the stated hyperparameter condition | AdamW on PTB · PTB | parameter ℓ∞ norm | Hurt |
| Use a batch size equal to half the training dataset | AdamW on PTB · PTB, half-dataset batch | parameter ℓ∞ norm | No clear effect |
| Use ℓ∞ geometry instead of ℓ2 geometry | Steepest descent and normalized steepest descent · Synthetic quadratic, 100 dimensions | optimization performance | Helped |
| Add weight decay | Normalized steepest descent with ℓ∞ or ℓ2 geometry · Synthetic quadratic, 100 dimensions | optimization performance | Helped |
Surprising findings
- With default beta1=0.9, beta2=0.999 (beta1 < beta2 with 1-beta2 < lambda*eta < 1-beta1), AdamW's l_inf norm is not bounded by 1/lambda, showing the beta1 <= beta2 condition matters.
- Constructed counterexample for beta1 > beta2 where AdamW converges outside the l_inf ball of radius 1/lambda.
Limitations
- Results are asymptotic and full-batch (deterministic); no non-asymptotic convergence rates for AdamW.
- Requires beta1 <= beta2 and non-increasing learning rate with divergent sum; the l_inf bound is only tight in these regimes.
- Empirical validation is limited to a small two-layer transformer on PTB and one synthetic problem.
Experimental setup
Datasets: Penn Treebank (PTB) synthetic 100-dimensional quadratic loss
Benchmarks: Penn Treebank (PTB) language modeling
Metrics: l_inf norm of parameters training loss
Baselines: Adam with l2 regularization Normalized-GD SignGD Lion-K
Builds on: Adam AdamW SignGD Frank-Wolfe Lion-K
Closest papers by meaning
- On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm (NeurIPS 2025 · similarity 0.90)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.90)
- On the Implicit Bias of Adam (ICML 2024 · similarity 0.88)
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.88)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.87)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.