Research map / Architectures and efficiency / RNNs and normalization
Learned Optimizers that Scale and Generalize
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned optimizer instead of RMSProp or ADAM | Meta-training corpus problems · Meta-training set | training loss | Helped |
| Use the learned optimizer instead of ADAM, RMSProp, or SGD with momentum | Learned optimizer · Small convolutional and fully connected neural networks | final loss and iterations to convergence | No clear effect |
| Apply the learned optimizer to large neural networks despite training only on small non-neural-network problems | Learned optimizer · ImageNet, Inception V3 and ResNet V2 | training loss: 10K to 20K steps; approximately 6.5 | Mixed |
| Remove attention (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Hurt |
| Remove momentum on multiple timescales (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Hurt |
| Remove dynamic input scaling (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Hurt |
| Remove relative learning rate (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Hurt |
| Replace the average function-value meta-objective with a log meta-objective (ablation) | Full learned optimizer · Meta-training objective | optimizer convergence | Helped |
| Use trainable initial weights for the RNN (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Helped |
| Use a learned accumulation decay for multiple gradient timescales (ablation) | Full learned optimizer · MNIST, 2-layer convolutional network | training loss | Helped |
| Remove the Global RNN (ablation) | Hierarchical learned optimizer · Simple test problems | performance | Hurt |
Closest papers by meaning
- Towards Learned Optimization Free Lunch (ICLR 2026 · similarity 0.91)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.89)
- Exploring Landscapes for Better Minima along Valleys (NeurIPS 2025 · similarity 0.88)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.88)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.87)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.