Research map / Optimization / Optimizers
QLABGrad: A Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QLABGrad, a hyperparameter-free learning rate scheme that fits a quadratic to the loss along the gradient direction using one extra forward pass, with a convergence proof under Lipschitz-smooth loss.
Problem
Choosing learning rates for deep network training typically requires empirical tuning or trial and error; existing adaptive schemes add hyperparameters or lack convergence guarantees.
Approach
Approximate the loss along the negative gradient direction by a quadratic in the step size, determined by the current loss, gradient norm, and loss at one pre-learning rate; the minimizer gives the learning rate, with a fallback to the pre-learning rate when it is negative, and a convergence guarantee under M-Lipschitz smoothness.
Key results
- On MNIST, QLABGrad reaches 99.18% test accuracy on CNN, versus 99.09% for SGD and 97.51% for Adam.
- QLABGrad achieves lower training loss than competing schemes on CIFAR-10 with ResNet-18 and on Tiny-ImageNet, per the authors' loss curves.
- Per-iteration training time is about 1.4x SGD, compared with about 1.3x for HGD and L4 and 1.6x for LQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QLABGrad instead of LQA | MLP · MNIST test | accuracy: 98.47 -> 98.36 | Hurt |
| Use QLABGrad instead of SGD | CNN · MNIST test | accuracy: 99.09 -> 99.18 | Helped |
| Use QLABGrad instead of HGD | CNN · MNIST test | accuracy: 99.06 -> 99.18 | Helped |
| Use QLABGrad instead of the other tested optimizers | CNN · MNIST | training loss | Helped |
| Use QLABGrad instead of the compared schemes | ResNet-18 · CIFAR-10 | training loss | Helped |
| Use QLABGrad instead of the compared methods | ResNet-18 · Tiny-ImageNet | training loss | Helped |
| Use QLABGrad instead of SGD | deep learning training · MNIST, CIFAR-10, and Tiny-ImageNet | training time: about 1.4 times | Hurt |
| Use QLABGrad with FindPLR instead of relying on a fixed, potentially suboptimal pre-learning rate | QLABGrad · deep learning tasks | training loss | Helped |
Surprising findings
- RMSProp and Adam with initial learning rate 0.1 diverge on MNIST MLP and CNN, so their loss curves are out of range.
- Adagrad performs well on MNIST but slower on Tiny-ImageNet, showing inconsistency across datasets.
Limitations
- Convergence proof requires M-Lipschitz smooth loss, which is hard to verify for deep networks; the pre-learning rate procedure only bounds it by 2/M.
- Per-step overhead of about 1.4x SGD wall-clock time.
- Experiments are limited to small/medium benchmarks (MNIST, CIFAR-10, Tiny-ImageNet) with ResNet-18 and small MLP/CNN, with no large-scale ImageNet results despite the abstract.
Experimental setup
Datasets: MNIST CIFAR-10 Tiny-ImageNet
Benchmarks: MNIST CIFAR-10 Tiny-ImageNet
Metrics: test accuracy training loss learning rate trajectories wall-clock time overhead
Baselines: SGD RMSProp Adagrad Adam HGD L4GD LQA E-decay SS-decay CA-decay
Builds on: Box-Kanemasu method L4 LQA Adam hypergradient descent
Closest papers by meaning
- Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation (ICLR 2022 · similarity 0.86)
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · similarity 0.85)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.85)
- AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining (NeurIPS 2025 · similarity 0.84)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.83)
- QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.