Research map / Learning theory and algorithms / Combinatorial and search optimization
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune only the learning rate rather than all Adam hyperparameters (ablation) | Adam · Classification tasks, budgets up to 16 trials | average performance | Helped |
| Tune all Adam hyperparameters instead of only the learning rate (ablation) | Adam · CIFAR-100, FMNIST, and IMDb at higher budgets | average performance | Helped |
| Tune momentum and weight decay rather than fixing them (ablation) | SGD · Classification tasks, budgets up to 16 trials | average performance | Hurt |
| Tune additional hyperparameters beyond the learning rate (ablation) | Optimizers · Regression tasks other than MNIST-VAE | performance | Mixed |
| Add Poly learning-rate decay (ablation) | SGD with fixed momentum and weight decay · CIFAR-10 | CPE accuracy (%): 81.0 -> 82.1 | Helped |
| Add Poly learning-rate decay (ablation) | Adam-LR · IMDb | CPE accuracy (%): 85.9 -> 86.0 | No clear effect |
| Parameterize SGD momentum through effective learning rate and sample learning rate/effective learning rate together (ablation) | SGD-MW · Classification tasks, low HPO budgets | relative performance | Helped |
| Parameterize SGD momentum through effective learning rate (ablation) | SGD-MW · Large HPO budgets | relative performance | Hurt |
| Tune all Adam hyperparameters rather than only the learning rate (ablation) | Adam · High HPO budgets across tasks | relative performance to the best optimizer: 2-3% | Helped |
| Use Adam-LR rather than SGD variants | Optimizer selection · Early HPO budgets across tasks | relative performance: 20% | Helped |
Closest papers by meaning
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers (ICML 2021 · similarity 0.94)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.88)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.87)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.87)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.