Research map / Optimization / Optimizers
Learning Gradient Descent: Better Generalization and Longer Horizons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RNNprop instead of DMoptimizer | learned optimizer on Base MLP with ReLU activation · Base MLP, ReLU | final loss: 1.05 -> 0.27 | Helped |
| Use RNNprop instead of Adam | learned optimizer on Base MLP with ELU activation · Base MLP, ELU | final loss: 0.26 -> 0.24 | Helped |
| Use RNNprop instead of Adam | learned optimizer on Base MLP with tanh activation · Base MLP, tanh | final loss: 0.31 -> 0.28 | Helped |
| Train RNNprop with two-layer LSTM optimizees | RNNprop · LSTM sequence prediction, 2 layers | final loss: 0.44 -> 0.28 | Helped |
| Train RNNprop with smaller-noise LSTM data | RNNprop · LSTM sequence prediction, small noise | final loss: 0.39 -> 0.36 | Helped |
| Remove convex-function combination (ablation) | RNNprop trained for 15k iterations · Base MLP, sigmoid | final loss: 0.30 -> 0.33 | Hurt |
| Remove convex-function combination (ablation) | RNNprop trained for 10k iterations · Base MLP, ReLU | final loss: 0.29 -> 0.31 | Hurt |
| Remove convex-function combination (ablation) | RNNprop trained for 15k iterations · Base MLP, ReLU | final loss: 0.27 -> 0.32 | Hurt |
| Remove convex-function combination (ablation) | RNNprop trained for 5k iterations · Base MLP, sigmoid | final loss: 0.33 -> 0.33 | No clear effect |
| Remove Random Scaling (ablation) | RNNprop · Base MLP with ReLU, 1000 optimization steps | final loss | Hurt |
| Add Random Scaling and convex-function combination to DMoptimizer training (ablation) | DMoptimizer · Base MLP with ReLU, 1000 optimization steps | final loss | Mixed |
| Train RNNprop with convex-function combination (ablation) | RNNprop training · Base MLP and Base MLP with ReLU | final loss: Base MLP 10k: 0.31 vs 0.32; ReLU 15k: 0.27 vs 0.32 | Helped |
Closest papers by meaning
- Learned Optimizers that Scale and Generalize (ICML 2017 · similarity 0.91)
- DeepOBS: A Deep Learning Optimizer Benchmark Suite (ICLR 2019 · similarity 0.90)
- Diving into the shallows: a computational perspective on large-scale shallow learning (NeurIPS 2017 · similarity 0.90)
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization (ICLR 2020 · similarity 0.90)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.88)
- How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.