Research map / Optimization / Optimizers
On the Variance of the Adaptive Learning Rate and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rectify adaptive learning-rate variance instead of using heuristic warmup | Adam on IWSLT'14 DE-EN · IWSLT'14 DE-EN | BLEU: 34.66 ± 0.014 -> 34.76 ± 0.003 | Helped |
| Rectify adaptive learning-rate variance instead of using heuristic warmup | Adam on IWSLT'14 EN-DE · IWSLT'14 EN-DE | BLEU: 28.56 ± 0.067 -> 28.48 ± 0.054 | Mixed |
| Rectify adaptive learning-rate variance instead of using heuristic warmup | Adam on WMT'16 EN-DE · WMT'16 EN-DE | BLEU: 27.03 -> 27.27 | Helped |
| Use RAdam rather than Adam with warmup | Adam on CIFAR10 · CIFAR10 test set | accuracy: 91.29 -> 91.38 | No clear effect |
| Use RAdam rather than Adam with 100 warmup steps | Adam, learning rate 0.1 · CIFAR10 test set | accuracy: 90.13 -> 91.06 | Helped |
| Replace RAdam with Adam on the first 4 updates (ablation) | RAdam · CIFAR10, learning rate 0.1 | test accuracy: 91.08 -> 89.98 | Hurt |
| Use SGD for the first 4 updates and Adam for updates 5-8 (ablation) | RAdam · CIFAR10, learning rate 0.1 | test accuracy: 91.08 -> 90.29 | Hurt |
| Collect 2000 adaptive-rate samples before updating parameters | Adam (Adam-2k) · IWSLT'14 German-to-English | training loss: around 10 -> below 3 | Helped |
| Increase epsilon to 10^-4 | Adam-eps · IWSLT'14 German-to-English | training performance | Mixed |
| Use Adam without warmup | Adam on IWSLT'14 German-to-English · IWSLT'14 German-to-English | training loss: 3 -> around 10 | Hurt |
| Rectify adaptive learning-rate variance instead of vanilla Adam | Adam on One Billion Word, CIFAR10, and ImageNet · One Billion Word, CIFAR10, ImageNet | performance | Helped |
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.88)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.88)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.87)
- A Method for Enhancing Generalization of Adam by Multiple Integrations (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.