Research map / Optimization / Optimizers
Understanding and correcting pathologies in the training of learned optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the combined reparameterization and evolutionary-strategies gradient estimator instead of either estimator alone (ablation) | Learned optimizer outer-training · CNN optimization tasks | outer-loss performance and training stability | Helped |
| Use only the reparameterization gradient estimator (ablation) | Learned optimizer outer-training · CNN optimization tasks | outer-loss performance | Hurt |
| Use only evolutionary strategies (ablation) | Learned optimizer outer-training · CNN optimization tasks | outer-loss performance | Hurt |
| Use a fixed truncation length of 10 steps rather than an increasing unroll schedule (ablation) | Learned optimizer outer-training · CNN optimization tasks | outer-loss performance | Hurt |
| Use a fixed truncation length of 100 steps rather than an increasing unroll schedule (ablation) | Learned optimizer outer-training · CNN optimization tasks | outer-loss performance | Hurt |
| Use a fixed truncation length of 1000 steps rather than an increasing unroll schedule (ablation) | Learned optimizer outer-training · CNN optimization tasks | training variance | Hurt |
| Remove momentum-based features from the optimizer inputs (ablation) | Learned optimizer · CNN optimization tasks | convergence | Hurt |
| Remove the current iteration number from the optimizer inputs (ablation) | Learned optimizer · CNN optimization tasks | outer-loss performance | No clear effect |
| Train the learned optimizer to minimize training loss rather than validation loss | Learned optimizer · Outer-test CNN tasks | test loss | Hurt |
| Use the learned optimizer rather than tuned first-order methods | Learned optimizer · Outer-test CNN tasks | test loss | Helped |
| Use the learned optimizer rather than the tuned 8-parameter Adam+Reg+Decay baseline | Learned optimizer · Outer-test tasks | training and test loss | Helped |
| Use the learned optimizer rather than Adam, SGD, or the learned optimizer's competitors | CNN inner-loop training on CPU · CNN inner-loop training, CPU | batches per second: 80 vs 92; 80 vs 93 | Hurt |
Closest papers by meaning
- Learned Optimizers that Scale and Generalize (ICML 2017 · similarity 0.91)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.90)
- Reverse engineering learned optimizers reveals known and novel mechanisms (NeurIPS 2021 · similarity 0.90)
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers (ICML 2021 · similarity 0.89)
- A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases (NeurIPS 2022 · similarity 0.89)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.