Research map / Optimization / Optimizers
ADOPT: Modified Adam Can Converge with Any $\beta_2$ with the Optimal Rate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADOPT, a modified Adam that removes the current gradient from the second-moment estimate and reorders momentum update and normalization, giving O(1/sqrt(T)) nonconvergence-free convergence for any beta2 without bounded-noise assumptions.
Problem
Adam does not converge in general unless beta2 is chosen problem-dependently; existing fixes (AMSGrad) need an impractical uniformly bounded gradient noise assumption.
Approach
Decorrelate the second-moment estimate from the current gradient by normalizing with v_{t-1} rather than v_t, and change the order of momentum update and normalization so the momentum only accumulates already-normalized gradients; this yields convergence for any beta2 under a bounded second-moment assumption.
Key results
- On ImageNet with Swin Transformer-tiny at 300 epochs, ADOPT reaches 81.50% top-1 vs 81.26% for AdamW.
- On MNIST NVAE density estimation at 300 epochs, ADOPT achieves 78.88 NLL vs 79.41 for Adamax (lower is better).
- LLaMA-7B instruction tuning: MMLU score 42.13 with ADOPT vs 41.2 with Adam (base 35.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADOPT instead of AdamW | SwinTransformer-tiny · ImageNet, 200 epochs | top-1 accuracy (%): 79.29 ± 0.05 -> 79.62 ± 0.03 | Helped |
| Use ADOPT instead of AdamW | SwinTransformer-tiny · ImageNet, 300 epochs | top-1 accuracy (%): 81.26 ± 0.04 -> 81.50 ± 0.04 | Helped |
| Use ADOPT instead of Adamax | NVAE · MNIST density estimation, 200 epochs | negative log-likelihood: 80.19 ± 0.08 -> 79.02 ± 0.10 | Helped |
| Use ADOPT instead of Adamax | NVAE · MNIST density estimation, 300 epochs | negative log-likelihood: 79.41 ± 0.07 -> 78.88 ± 0.09 | Helped |
| Use ADOPT instead of Adam | LLaMA-7B fine-tuning · MMLU | score: 41.2 -> 42.13 | Helped |
| Replace Adam with ADOPT | GPT-2 pretraining · OpenWebText, batch size 96 | training stability | Helped |
| Replace Adam with ADOPT | GPT-2 pretraining · OpenWebText, large batch size | training loss | Helped |
| Use ADOPT instead of Adam | ResNet-18 · CIFAR-10 classification | test accuracy | Helped |
| Use ADOPT instead of Adam | MLP · MNIST classification | accuracy | Helped |
| Add decorrelation by excluding the current gradient from the second-moment estimate (ablation) | ADOPT · Toy optimization, k = 50 | convergence | Helped |
| Change the order of momentum update and normalization (ablation) | ADOPT · Toy optimization, k = 50 | convergence | Helped |
| Use both decorrelation and the changed momentum-normalization order rather than removing either change (ablation) | ADOPT · Toy optimization, k = 50 | convergence | Helped |
Surprising findings
- ADOPT can fail to converge under without-replacement sampling in a constructed finite-sum counterexample, a caveat the authors note.
- Adam causes loss spikes and fails to converge in GPT-2 pretraining with small batch size, while ADOPT trains stably.
Limitations
- Analysis still assumes uniformly bounded second moment of stochastic gradient (Assumption on E||g||^2 <= G^2), not the weaker bounded-variance assumption.
- Convergence fails in a without-replacement sampling counterexample, so with-replacement sampling is recommended for small datasets.
- Vanilla ADOPT can be unstable when near-zero second-moment entries occur, motivating a clipped variant with its own theory.
Experimental setup
Datasets: MNIST CIFAR-10 ImageNet OpenWebText Stanford Alpaca
Benchmarks: MNIST CIFAR-10 ImageNet MMLU
Metrics: test accuracy top-1 accuracy negative log-likelihood validation loss convergence of objective in toy problem
Baselines: Adam AMSGrad AdaShift AdamW Adamax RMSprop
Builds on: Adam RMSprop AMSGrad AdaShift
Closest papers by meaning
- Convergence of Adam Under Relaxed Assumptions (NeurIPS 2023 · similarity 0.90)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.86)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.86)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.86)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.