Research map / Optimization / Optimizers
Reverse engineering learned optimizers reveals known and novel mechanisms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned optimizer instead of tuned momentum, RMSProp, or Adam | Linear regression · Linear regression | meta-objective | Helped |
| Use the learned optimizer instead of tuned momentum, RMSProp, or Adam | Rosenbrock · Rosenbrock | meta-objective | Helped |
| Use the learned optimizer instead of tuned momentum, RMSProp, or Adam | Two moons · Two moons | meta-objective | Helped |
| Use the learned optimizer instead of tuned momentum, RMSProp, or Adam | MNIST CNN · MNIST classification | meta-objective | Helped |
| Use multiple momentum-like state modes | Learned optimizers · Linear regression, Rosenbrock, Two moons, and MNIST | effective learning rate and momentum dynamics | Mixed |
| Saturate the update function for large gradients | Learned optimizers · Rosenbrock | update magnitude for large gradients | Helped |
| Use autonomous state dynamics to vary the learning rate over time | Learned optimizers · Linear regression and MNIST | effective learning rate | Mixed |
| Reduce the learning rate after encountering large gradients | Learned optimizers · Linear regression, Rosenbrock, and Two moons | effective learning rate | Helped |
| Use a fixed-point manifold to adapt learning rates across parameter trajectories | MNIST learned optimizer · MNIST classification | effective learning rate | Mixed |
| Tune learning rates by parameter type and layer | MNIST learned optimizer · MNIST classification | effective learning rate | Mixed |
Closest papers by meaning
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.92)
- Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules (NeurIPS 2025 · similarity 0.87)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.87)
- Understanding Optimization in Deep Learning with Central Flows (ICLR 2025 · similarity 0.87)
- Interpreting Neural Combinatorial Optimization via Evolving Programmatic Bottlenecks (NeurIPS 2026 · similarity 0.86)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.