Research map / Optimization / Optimizers
Gradient Descent: The Ultimate Optimizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply online hyperoptimization to SGD's step size | SGD · MNIST | test error: 8.99±0.05% -> 4.81±0.10% | Helped |
| Use Adam instead of SGD to optimize SGD's step size | SGD · MNIST | test error: 8.99±0.05% -> 4.86±0.06% | Helped |
| Optimize Adam hyperparameters online with SGD | Adam · MNIST | test error: 4.67±0.06% -> 3.03±0.02% | Helped |
| Optimize only Adam's learning rate, rather than all hyperparameters (ablation) | Adam / SGD(10^-5) · MNIST | test error: 3.03±0.02% vs 3.12±0.04%; 3.05±0.09% vs 3.04±0.08% | Mixed |
| Optimize AdaGrad's hyperparameters online | AdaGrad · MNIST | test error: 7.40±0.08% -> 6.90±0.16% | Helped |
| Use AdaGrad to optimize AdaGrad's hyperparameters | AdaGrad · MNIST | test error: 7.40±0.08% -> 5.03±0.23% | Helped |
| Optimize RMSProp's learning rate and gamma instead of only its learning rate (ablation) | RMSProp / SGD(10^-4) · MNIST | test error: 3.55±0.23% -> 3.33±0.07% | Helped |
| Adapt the initial learning rate online | HyperAdam · Tolstoy Char-RNN, expert learning rate | perplexity: 5.41±0.26 vs 5.27±0.31 | No clear effect |
| Adapt from a too-high initial learning rate | HyperAdam · Tolstoy Char-RNN | perplexity: 5.77±0.34 -> 5.45±0.76 | Helped |
| Adapt from a too-low initial learning rate | HyperAdam · Tolstoy Char-RNN | perplexity: 8.71±0.91 -> 6.51±0.88 | Helped |
| Increase the height of the recursive hyperoptimizer stack | Hyperoptimizers · MNIST, CIFAR-10, and Tolstoy | robustness to initial hyperparameters | Helped |
| Use a height-1 hyperoptimizer from alpha=10^-6 | ResNet-152 fine-tuning · Caltech-256 | error: 91.5% -> ≈ 11% | Helped |
Closest papers by meaning
- Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation (ICLR 2022 · similarity 0.90)
- Learning Gradient Descent: Better Generalization and Longer Horizons (ICML 2017 · similarity 0.88)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.88)
- Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (ICML 2020 · similarity 0.88)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.87)
- LPGD: A General Framework for Backpropagation through Embedded Optimization Layers (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.