Research map / Optimization / Optimizers
On the Adequacy of Untuned Warmup for Adaptive Optimization
Jerry Ma, Denis Yarats
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RAdam with untuned exponential or linear warmup in Adam | Adam on ResNet-50 · ImageNet | top-1 error: 28.2 % ± 0.1; 28.3 % ± 0.1; 28.4 % ± 0.1 | No clear effect |
| Replace RAdam with untuned exponential or linear warmup in Adam | Adam on Transformer LM · WIKITEXT-103 | validation perplexity: 20.0 ± 0.0; 20.0 ± 0.0; 20.1 ± 0.1 | No clear effect |
| Replace RAdam with untuned exponential or linear warmup in Adam | Adam on Transformer · WMT16 EN-DE | BLEU: 26.2 ± 0.2; 26.4 ± 0.0; 26.4 ± 0.0 | No clear effect |
| Vary the first four RAdam iterations among doing nothing, Adam at the fifth-step rate, or gradually warming up to that rate (ablation) | RAdam · Experimental settings described in the paper | practical performance | No clear effect |
| Replace the adaptive variance-based account of warmup with analysis of actual Adam update magnitudes | Adam warmup analysis · Simulated local minimum and EMNIST | update magnitude: 1 · α; ≈ 0.16 · α; 0.153 · α; around 0.15 · α | Mixed |
Closest papers by meaning
- Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers (NeurIPS 2026 · similarity 0.89)
- Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.88)
- Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (NeurIPS 2024 · similarity 0.88)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.86)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
