Research map / Architectures and efficiency / RNNs and normalization
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Try several optimizers with default hyperparameters instead of tuning one fixed optimizer | Deep learning optimization on the benchmark workloads · Eight DeepOBS problems | final test performance | No clear effect |
| Increase the tuning budget from default parameters to the medium budget without a schedule | The benchmarked optimizers · Eight DeepOBS problems, median across settings | relative final test performance: roughly $3.4\,\%$ | Helped |
| Use a trapezoidal learning-rate schedule with the large tuning budget | The benchmarked optimizers · Eight DeepOBS problems, median across settings | relative final test performance: roughly $5.2\,\%$ | Helped |
| Apply an additional untuned learning-rate schedule | The benchmarked optimizers · Eight DeepOBS problems | final test performance | Mixed |
| Use Adam as a general-purpose optimizer | Deep learning optimization · Eight DeepOBS problems | final test performance | No clear effect |
| Use NAG instead of Adam | Deep learning optimization · Image classification tasks | final test performance | Helped |
| Use RMSProp | Deep learning optimization · Natural language modeling (RNN) | final test performance | Helped |
| Use RMSProp | Deep learning optimization · RNN | final test performance | Helped |
| Use NAG | Deep learning optimization · All-CNN-C | final test performance | Helped |
| Choose the best of untuned Adam and AdaBound | Deep learning optimization · Benchmark problems excluding the two VAE problems | final test performance | Helped |
| Use newer optimizers in place of established methods | Deep learning optimization · Eight DeepOBS problems | final test performance | No clear effect |
| Use default parameters for Momentum, NAG, SGD, AMSGrad, or Adadelta rather than tuning | These optimizers · Several DeepOBS problems | final test performance | Hurt |
Closest papers by meaning
- Exploring Landscapes for Better Minima along Valleys (NeurIPS 2025 · similarity 0.88)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.88)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.88)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.