Research map / Optimization / Optimizers
Quasi-hyperbolic momentum and Adam for deep learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NAG with QHM | ResNet152 · ImageNet (ILSVRC2012) | validation top-1 error rate: 0.2137±0.0011 -> 0.2128±0.0005 | Helped |
| Replace NAG with QHM | FConvLM · WikiText-103 | validation perplexity: 34.92±0.33 -> 34.45±0.17 | Helped |
| Replace Adam with QHAdam | TD3 · MuJoCo environments | average reward | Mixed |
| Use QHAdam instead of Adam, including immediate discounting of the second moment (ν₂ = 0.7) | Transformer · WMT16 EN-DE | validation BLEU: 29.17±0.07 -> 29.45±0.06 | Helped |
| Use QHAdam instead of Adam, including immediate discounting of the second moment (ν₂ = 0.7) | Transformer · WMT16 EN-DE | training convergence across seeds | Helped |
| Increase β₁ substantially | Adam and QHAdam · MuJoCo environments | average reward | No clear effect |
| Use higher-β₁ QHAdam parameterizations | Transformer · WMT16 EN-DE | validation BLEU | No clear effect |
Closest papers by meaning
- Understanding the Role of Momentum in Stochastic Gradient Methods (NeurIPS 2019 · similarity 0.90)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.88)
- Better SGD using Second-order Momentum (NeurIPS 2022 · similarity 0.88)
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.87)
- Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · similarity 0.87)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.