Research map / Optimization / Optimizers
Towards Better Generalization of Adaptive Gradient Methods
Yingxue Zhou, Belhal Karimi, Jinxing Yu, Zhiqiang Xu, Ping Li
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAGD instead of RMSprop/Adam/AdaBound/SGD | LSTM optimizer · Penn Treebank 2-layer LSTM | test perplexity: 65.96 -> 60.66 | Improved |
| Use SAGD instead of best baseline AdaBound | LSTM optimizer · Penn Treebank 3-layer LSTM | test perplexity: 62.33 -> 59.43 | Improved |
| Use SAGD instead of Adam/RMSprop | LSTM optimizer · Penn Treebank 3-layer LSTM | overfitting after 150 epochs | Improved |
| Use SAGD instead of RMSprop/Adam | 2-layer LSTM · Penn Treebank | train perplexity | Hurt |
| Use SAGD instead of baselines | 2-layer ReLU network · MNIST | test accuracy | Improved |
| Use SAGD instead of baselines | 2-layer Sigmoid network · MNIST | test accuracy | Improved |
| Use SAGD instead of baselines | ResNet-18 · CIFAR-10 | test accuracy | Improved |
| Use SAGD instead of adaptive baselines | VGG-19 · CIFAR-10 | test loss | Mixed |
| Use SAGD instead of baselines | biLSTM · SNLI | train vs test perplexity | Mixed |
Closest papers by meaning
- The Marginal Value of Adaptive Gradient Methods in Machine Learning (NeurIPS 2017 · similarity 0.90)
- Adaptive Methods for Nonconvex Optimization (NeurIPS 2018 · similarity 0.89)
- The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks (ICML 2021 · similarity 0.88)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.87)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.86)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
