Research map / Optimization / Optimizers
In Search of Adam’s Secret Sauce
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large empirical study (1,500+ LM runs) shows Adam beats signed-momentum, SGD and RMSprop variants on transformer LM pretraining, and that setting beta1=beta2 gives near-optimal Adam, which admits a mean-variance online variational inference interpretation.
Problem
Why Adam outperforms simplified variants (SignSGD, Signum, SGD with clipping, RMSprop) when training transformer language models, and which Adam hyperparameter simplification preserves its performance.
Approach
Empirically identifies beta1=beta2 as a near-optimal, one-parameter Adam simplification, and shows that under this constraint the second-moment denominator equals an online estimate of gradient variance, so Adam is an online Gaussian mean-field variational inference method and an SNR-modulated (mollified) signed-momentum update.
Key results
- At 160M parameters on SlimPajama (3.2B tokens), Adam reaches 21.86 validation perplexity versus 23.23 for Signum, 27.04 for RMSprop and 53.62 for SGD; Signum closes 96% of the SGD-Adam gap yet is about 25% slower.
- Across 500 runs, setting beta1=beta2 gives near-optimal Adam performance and lower loss correlates with smaller |beta1 - beta2|.
- Adam with beta1=beta2 equals a signed momentum method with an SNR-based adaptive mollifier; the variance term is precisely an online variance estimate only when beta1=beta2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace tuned SGD with tuned Adam | language modeling on SlimPajama-627B · SlimPajama-627B validation (160M, 3.2B tokens, sequence length 2048, batch size 256) | validation perplexity: 53.62 ± 5.14 -> 21.86 ± 0.21 | Helped |
| replace Adam with Signum | language modeling on SlimPajama-627B · SlimPajama-627B validation (160M, 3.2B tokens, sequence length 2048, batch size 256) | validation perplexity: 21.86 ± 0.21 -> 23.23 ± 0.16 | Hurt |
| set Adam's momentum parameters equal, β1=β2 | Adam · SlimPajama-627B and evaluated language-model settings | validation perplexity | Helped |
| make β1 and β2 closer | Adam · 160M SlimPajama language models | loss | Helped |
| increase sequence length from 512 to 2048 | Adam relative to Signum · SlimPajama, 160M | validation perplexity: 0.7 validation perplexity | Mixed |
| set epsilon within 1e-6 to 1e-15 rather than 1e-8 (ablation) | AdamW · SlimPajama-627B validation (160M) | validation perplexity: 21.56 ± 0.19; 21.86 ± 0.21; 21.87 ± 0.04; 21.89 ± 0.2; 21.91 ± 0.18 | No clear effect |
| remove bias correction (ablation) | Adam with zero-initialized moving averages · SlimPajama-627B validation (160M) | validation perplexity: 21.86 ± 0.21 -> 21.89 ± 0.16 | No clear effect |
| initialize Adam moving averages from the first gradient instead of zero, without bias correction (ablation) | Adam · SlimPajama-627B validation (160M) | validation perplexity: 21.86 ± 0.21 -> 22.58 ± 0.35 | Hurt |
| zero-initialize Signum momentum rather than initialize it from the first gradient (ablation) | Signum · SlimPajama-627B validation (160M) | validation perplexity: 23.23 ± 0.16 -> 23.30 ± 0.25 | No clear effect |
| remove global norm clipping (ablation) | Signum · SlimPajama-627B language modeling (160M) | validation perplexity | Mixed |
| add a fixed epsilon mollifier to Signum (ablation) | Signum · SlimPajama-627B validation (160M) | validation perplexity | No clear effect |
| vary EMA initialization for Signum (ablation) | Signum · SlimPajama-627B validation (160M) | validation perplexity | No clear effect |
Surprising findings
- Signum, after extensive tuning, closes 96% of the perplexity gap between SGD and Adam but still results in a 25% effective slowdown relative to Adam.
- Fixed-constant epsilon in place of the adaptive variance term does not reproduce Adam's adaptive effect, and epsilon in Adam is largely irrelevant in the tested range.
Limitations
- Results depend on the fixed hyperparameter grids, though the authors show optima lie inside the grids.
- At small batch sizes, the beta1=beta2 advantage shows a slight shift.
- The variational inference view does not explain why the moment estimates are combined into the specific quotient used by Adam.
Experimental setup
Datasets: SlimPajama-627B Fineweb
Benchmarks: SlimPajama validation perplexity (100M held-out tokens) Fineweb validation perplexity
Metrics: validation perplexity mean and 2-sigma interval across seeds
Baselines: SGD SGD with clipping Signum SignSGD RMSprop EMASign
Builds on: Adam signSGD Signum AdamW Balles and Hennig 2018
Closest papers by meaning
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.91)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.89)
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.88)
- Understanding Adam Requires Better Rotation Dependent Assumptions (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.