Research map / Optimization / Optimizers
Learning to learn by gradient descent by gradient descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn an LSTM-based update rule on sampled 10-dimensional quadratic problems | Optimization of 10-dimensional quadratic functions · 10-dimensional quadratic functions | objective loss | Helped |
| Learn an LSTM-based optimizer rather than use hand-designed optimizers | MNIST MLP with one hidden layer and 20 sigmoid units · MNIST | cross entropy | Helped |
| Continue using the optimizer trained for 100 steps through 200 steps | MNIST MLP · MNIST, 200 optimization steps | cross entropy | Helped |
| Transfer the optimizer to an MLP with 40 hidden units instead of 20 | LSTM optimizer trained on the 20-unit MNIST MLP · MNIST, 40 hidden units | cross entropy | Helped |
| Transfer the optimizer to a network with two hidden layers instead of one | LSTM optimizer trained on the one-layer MNIST MLP · MNIST, two hidden layers | cross entropy | Helped |
| Replace sigmoid activations with ReLU activations | LSTM optimizer trained on the sigmoid MNIST MLP · MNIST, ReLU MLP | cross entropy | Hurt |
| Use gradient preprocessing instead of only rescaling inputs by a constant (ablation) | LSTM optimizer · Optimization problems tested in the paper | optimization performance | Helped |
| Use two coordinatewise LSTMs, separating fully connected and convolutional parameters (ablation) | LSTM optimizer for the CIFAR convolutional model · CIFAR-10 | classification objective | Helped |
| Transfer the CIFAR-trained optimizer to datasets containing subsets of labels | LSTM optimizer trained on CIFAR-10 · CIFAR label subsets | classification objective | Helped |
| Train LSTM-sub on a disjoint subset of labels and transfer it to the other subset | LSTM-sub · CIFAR label subsets | classification objective | Helped |
| Transfer the Neural Art optimizer to a different style and higher resolution | LSTM optimizer trained with one style at 64x64 · Neural Art, new style and 128x128 resolution | style-transfer objective | Helped |
Closest papers by meaning
- How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks (NeurIPS 2024 · similarity 0.87)
- GradMetaNet: An Equivariant Architecture for Learning on Gradients (NeurIPS 2025 · similarity 0.85)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.85)
- Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context (ICML 2024 · similarity 0.85)
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · similarity 0.84)
- LPGD: A General Framework for Backpropagation through Embedded Optimization Layers (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.