Research map / Optimization / Implicit bias and generalization
Stepping on the Edge: Curvature Aware Learning Rate Tuners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that classical learning rate tuners (linesearch, quadratically greedy) undershoot the edge of stability and underperform constant rates in full-batch training, and proposes CDAT, a curvature-aware tuner that targets the edge of stability and matches or beats tuned constant rates and warm-up-like behavior.
Problem
Learning rate tuners built on greedy loss decrease (linesearch, quadratically greedy rules) are known to underperform in deep learning, and the reason is unclear given that sharpness undergoes complex dynamics (progressive sharpening then edge-of-stability stabilization).
Approach
Classical greedy tuners suppress the sharpness-stabilizing feedback of the edge of stability, causing a snowball of rising sharpness and falling learning rate. CDAT sets the learning rate to a scaled quadratic-approximation edge value (sigma times n_t/d_t) so that the optimizer operates at or slightly above the edge of stability, restoring this feedback.
Key results
- CDAT with sigma=2 (on edge) matches or beats fine-tuned constant learning rates across full-batch settings, while the quadratically greedy rule (sigma=1) underperforms, and it is always better than the greedy variant.
- Classical tuners (linesearch, quadratically greedy, hyper-gradient) show learning rates decreasing by up to about 3 orders of magnitude while sharpness keeps increasing, so their sharpness times learning rate stays below the stability threshold of 2.
- In the mini-batch regime, the optimal CDAT scaling factor shifts with batch size, and a linesearch can perform on par with or better than constant learning rates, suggesting stochasticity confounds prior success of tuners.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linesearch or quadratically greedy learning rates instead of a tuned constant learning rate | Gradient descent and RMSProp · Full-batch deep-learning benchmarks | training loss | Hurt |
| Set CDAT scaling near 2 instead of greedy scaling 1 (ablation) | CDAT · Full-batch deep-learning experiments | training loss | Helped |
| Increase CDAT scaling from 2 to 2.0625 (ablation) | CDAT · Full-batch deep-learning experiments | training loss | Helped |
| Increase CDAT scaling to 2.5 (ablation) | CDAT · Full-batch deep-learning experiments | training loss | Hurt |
| Use CDAT instead of a fine-tuned constant learning rate | Full-batch optimizers · Full-batch deep-learning experiments | training loss | Helped |
| Use CDAT instead of a prefixed learning-rate schedule | Full-batch optimizers · Full-batch deep-learning experiments | training loss | Hurt |
| Use CDAT with the exact Hessian sharpness as the base learning rate instead of the quadratic approximation (ablation) | CDAT-like sharpness rule · Full-batch experiments | training loss | Mixed |
| Add exponential moving average smoothing to the CDAT curvature estimates (ablation) | CDAT · Mini-batch experiments across batch sizes | performance stability | Helped |
| Decrease CDAT scaling as batch size decreases (ablation) | CDAT · Mini-batch experiments | performance | Helped |
| Use CDAT in the stochastic regime instead of tuned constant or scheduled learning rates | Gradient descent with momentum · Mini-batch experiments | performance | Mixed |
| Use CDAT instead of a tuned constant learning rate | Adam · Stochastic regime | performance | Mixed |
| Use Polyak stepsizes instead of CDAT at edge scaling | Gradient descent · Full-batch experiments | training loss | Helped |
Surprising findings
- Classical learning rate tuners that greedily minimize the loss in one step underperform constant learning rates in the full-batch regime, despite larger instantaneous decreases.
- Prefixed warm-up schedules can outperform CDAT, indicating the feedback loop does not capture all benefits of warm-up.
Limitations
- The simplified two-eigenvalue model predicts runaway sharpening for sigma<2 and divergence for sigma>2, missing higher-order stabilization.
- CDAT underperforms in the stochastic regime, especially with Adam, and the optimal scaling factor varies with batch size.
- CDAT does not fully capture the benefits of prefixed warm-up schedules.
Experimental setup
Datasets: MNIST CIFAR10 Tiny Shakespeare ImageNet-1K Imagenette
Benchmarks: CIFAR10 Tiny Shakespeare Imagenette ImageNet-1K MNIST
Metrics: train loss test error sharpness (largest Hessian eigenvalue) learning rate evolution gradient norm
Baselines: constant learning rate linesearch quadratically greedy rule Polyak stepsize hyper-gradient descent prefixed warm-up schedules
Builds on: edge of stability linesearch with Armijo condition quadratically greedy learning rate Polyak stepsizes hyper-gradient descent
Closest papers by meaning
- A Loss Curvature Perspective on Training Instabilities of Deep Learning Models (ICLR 2022 · similarity 0.90)
- Curvature Tuning: Provable Training-free Model Steering From a Single Parameter (NeurIPS 2025 · similarity 0.88)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
- Global curvature for second-order optimization of neural networks (ICML 2025 · similarity 0.86)
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares (NeurIPS 2025 · similarity 0.85)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.