Research map / Optimization / Optimizers
The Marginal Value of Adaptive Gradient Methods in Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGD instead of RMSProp | VGG+BN+Dropout · CIFAR-10 | top-1 test error: 9.60±0.19% -> 7.65±0.14% | Helped |
| Use SGD instead of adaptive methods | 2-Layer LSTM language model · War and Peace | test loss | Helped |
| Tune Adam's learning rate and decay rather than use its default configuration | Cross and Huang parser · Penn Treebank discriminative parsing development | labeled F1 | Helped |
| Use SGD instead of Adam | Cross and Huang parser · Penn Treebank discriminative parsing development | labeled F1: 91.11 -> 91.24 | Helped |
| Use a larger LSTM hidden dimension (1500 instead of 500) (ablation) | Choe and Charniak generative parser · Penn Treebank | development performance | Helped |
| Use dropout 0.7 instead of 0.5 (ablation) | Choe and Charniak generative parser · Penn Treebank | development performance | Helped |
| Use a more aggressive learning rate for adaptive methods | CIFAR-10 adaptive optimizers · CIFAR-10 | training and test performance | Hurt |
| Use RMSProp configurations that lower training loss below SGD | 2-Layer LSTM language model · War and Peace | test loss | Hurt |
| Use adaptive methods instead of non-adaptive methods | deep learning models evaluated in the paper · CIFAR-10, War and Peace, Penn Treebank | development/test performance | Mixed |
| Use Adam with the default configuration and no learning-rate decay instead of tuned Adam | Cross and Huang parser · Penn Treebank discriminative parsing | training performance | Helped |
Closest papers by meaning
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.91)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.89)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.