Research map / Optimization / Optimizers
Online Learning Rate Adaptation with Hypergradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add online hypergradient learning-rate updates to SGD | SGD · MNIST logistic regression | training and validation loss | Helped |
| Add online hypergradient learning-rate updates to SGD with Nesterov momentum | SGDN · MNIST logistic regression | training and validation loss | Helped |
| Add online hypergradient learning-rate updates to Adam | Adam · MNIST logistic regression | training and validation loss | Helped |
| Add online hypergradient learning-rate updates to SGD | SGD · MNIST multilayer neural network | training and validation loss | Helped |
| Add online hypergradient learning-rate updates to SGDN | SGDN · MNIST multilayer neural network | training and validation loss | Helped |
| Add online hypergradient learning-rate updates to Adam | Adam · MNIST multilayer neural network | training loss: an order of magnitude improvement | Helped |
| Add online hypergradient learning-rate updates to Adam | Adam · MNIST multilayer neural network | validation loss | Helped |
| Add online hypergradient learning-rate updates to SGD | SGD · CIFAR-10 VGG Net | validation loss | Helped |
| Add online hypergradient learning-rate updates to SGDN | SGDN · CIFAR-10 VGG Net | validation loss | Helped |
| Add online hypergradient learning-rate updates to Adam | Adam · CIFAR-10 VGG Net | validation loss | No clear effect |
| Add online hypergradient learning-rate updates to Adam | Adam · CIFAR-10 VGG Net | training loss | Helped |
| Use the transition extension that gradually replaces the adaptive rate with a fixed learning rate | Hypergradient descent · Preliminary experimental evaluation | convergence performance across hypergradient learning rates: a larger range of β | Helped |
Closest papers by meaning
- Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent (ICML 2025 · similarity 0.91)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.90)
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate (ICLR 2019 · similarity 0.89)
- On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization (ICLR 2019 · similarity 0.89)
- MoMo: Momentum Models for Adaptive Learning Rates (ICML 2024 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.