Research map / Optimization / Optimizers
Gradient descent with generalized Newton’s method
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes generalized Newton's method (GeN), which sets the learning rate in closed form as G^T g / g^T H g along any base optimizer's direction, estimating it from extra forward passes via quadratic fitting, and matches tuned LR schedulers across vision and language tasks.
Problem
Choosing learning rates for deep learning optimizers requires heavy tuning of schedulers, and full Hessian-informed second-order methods are too costly or incompatible with large-scale distributed training.
Approach
Derive the optimal learning rate along the optimizer's preconditioned direction g as G^T g / g^T H g, and estimate the two quantities from loss values at a few symmetric learning rates (quadratic curve fit / finite difference) using only forward passes, avoiding Hessian-vector products; applies on top of any base optimizer and recovers Newton's method when g = H^{-1}G.
Key results
- On ResNet50 image classification (5 epochs), GeN-SGD reaches 96.76 on CIFAR10 and 84.77 on CIFAR100, above the best constant/decay SGD baselines (e.g. 95.91 and 81.67).
- On GPT2 with LoRA on E2E, GeN achieves BLEU 67.30 versus 66.85 for linear decay.
- On GLUE with RoBERTa, GeN-based results are mixed versus published heuristic-scheduler numbers, improving on several tasks such as CoLA and RTE with full fine-tuning (RTE 80.5 vs 77.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SGD with GeN-SGD | ResNet50 · Image classification datasets | accuracy | Helped |
| Replace scheduled AdamW with GeN-AdamW | GPT2 with LoRA · E2E | BLEU: 66.85 -> 67.30 | Helped |
| Replace linear decay with GeN | GPT2 with LoRA · E2E | ROUGE-L: 67.78 -> 67.09 | Hurt |
| Replace manual learning-rate schedule with GeN | Mask R-CNN · Penn-Fudan object detection | AP: $0.802_{\pm0.025}$ -> $0.805_{\pm0.038}$ | No clear effect |
| Replace manual learning-rate schedule with GeN | Mask R-CNN · Penn-Fudan instance segmentation | AP: $0.768_{\pm0.022}$ -> $0.771_{\pm0.019}$ | No clear effect |
| Replace constant learning rate with GeN | BERT recommendation model · MovieLens-1m | Recall@1: 0.374 -> 0.405 | Helped |
| Update the learning rate every Φ>1 iterations (lazy update) (ablation) | GeN optimizers · Convergence (experiments across tasks) | convergence | No clear effect |
| Use larger batch sizes (ablation) | GeN optimizers · Training stability | convergence stability | Helped |
| Fit with five learning-rate points instead of three (ablation) | GeN · Optimization convergence | convergence stability | Helped |
| Apply smoothing to the learning-rate update (ablation) | GeN · Training experiments | training stability | Helped |
| Multiply the GeN learning rate by (1-t/T) (ablation) | GeN · Places365, INat2021, and GLUE | performance | Helped |
| Vary the initial learning rate over a wide range (ablation) | GeN-SGD · ResNet18 on CIFAR10 | final performance: 1000× | No clear effect |
Surprising findings
- The learning rate learned by GeN keeps increasing even when training is extended to 20 epochs, which heuristic schedulers do not capture.
- GeN-SGD is scale-invariant in the gradient, so it does not need gradient clipping; D-adaptation SGD diverged on all datasets with weight decay 5e-4.
Limitations
- Global convergence speed is not well understood; the learning rate is only locally optimal.
- Sub-sampling error from mini-batch loss estimation dominates and favors large batch sizes.
- Still requires an initial learning rate and a choice of symmetric finite-difference points; recommendation experiments needed larger batch size to converge stably.
Experimental setup
Datasets: CIFAR10 CIFAR100 Food101 GTSRB SVHN Places365 INat2021 E2E GLUE Penn-Fudan COCO MovieLens-1m
Benchmarks: CIFAR10 CIFAR100 Food101 GTSRB SVHN Places365 INat2021 E2E GLUE Penn-Fudan
Metrics: test accuracy BLEU ROUGE-L METEOR NIST CIDEr matthews correlation F1 average precision recall
Baselines: constant learning rate linear decay cosine decay Prodigy D-adaptation inverse sqrt schedule LoRA BitFit
Builds on: LQA D-adaptation Prodigy AdamW
Closest papers by meaning
- Learned Optimizers that Scale and Generalize (ICML 2017 · similarity 0.87)
- Gradient Regularized Newton Boosting Trees with Global Convergence (NeurIPS 2026 · similarity 0.87)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.85)
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · similarity 0.85)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
- ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.