Research map / Architectures and efficiency / RNNs and normalization
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use factored row-and-column second moments instead of full second-moment accumulators (ablation) | Adam on Transformer · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 25.4 | No clear effect |
| Approximate matrix second moments using row means only (ablation) | Adam on Transformer · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 25.2 | No clear effect |
| Approximate matrix second moments using column means only (ablation) | Adam on Transformer · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 0.3 | Hurt |
| Remove momentum (ablation) | Adam on Transformer · WMT 2014 En-De, newstest2013 | BLEU: 23.4 vs. 23.6; 23.1 vs. 0.1 | Mixed |
| Use constant fast second-moment decay β₂=0.9 rather than β₂=0.999 (ablation) | Adam without momentum · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 18.4; 0.1 -> 15.6 | Mixed |
| Add update clipping with d=1 (ablation) | Adam without momentum, no warmup · WMT 2014 En-De, newstest2013 | BLEU: 0.1 -> 21.5 | Helped |
| Use update clipping with d=2 instead of d=1 (ablation) | Adam without momentum, no warmup · WMT 2014 En-De, newstest2013 | BLEU: 21.5 -> 0.2 | Hurt |
| Use increasing second-moment decay 1−t⁻⁰·⁵ (ablation) | Adam without momentum · WMT 2014 En-De, newstest2013 | BLEU: 0.1 -> 21.1 | Helped |
| Use increasing second-moment decay 1−t⁻⁰·⁸ (ablation) | Adam without momentum · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 25.6; 0.1 -> 0.1 | Mixed |
| Use increasing second-moment decay 1−t⁻¹ (ablation) | Adam without momentum · WMT 2014 En-De, newstest2013 | BLEU: 25.6 -> 25.4; 0.1 -> 0.1 | Mixed |
| Add update clipping d=1 to increasing decay 1−t⁻⁰·⁸ (ablation) | Adam without momentum · WMT 2014 En-De, newstest2013 | BLEU: 0.1 -> 22.4 | Helped |
| Scale the step size relative to parameter RMS instead of using the absolute-step Adam scheme (ablation) | Adafactor versus Adam · WMT 2014 En-De, embedding σ=1/√d_model with multiplier 1 | BLEU: 24.2 -> 25.4 | Helped |
Closest papers by meaning
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.87)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.87)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.87)
- Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · similarity 0.86)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.86)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.