Research map / Optimization / Optimizers
Second-order and structured preconditioning
Builds diagonal, full-matrix, and Hessian-based preconditioners for adaptive optimization.
70 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Optimizers, keyword cluster “adaptive · optimizers · momentum”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 26 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.6% to 9.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Expected Tensor Decomposition with Stochastic Gradient Descent (AAAI 2016 · 41 citations)
- 2017 · Nonparametric Neural Networks (ICLR 2017 · 10 citations)
- 2018 · Adaptive Methods for Nonconvex Optimization (NeurIPS 2018 · 219 citations)
- 2019 · Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (CVPR 2019 · 81 citations)
- 2020 · Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (ICLR 2020 · 27 citations)
- 2021 · ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (AAAI 2021 · 190 citations)
- 2022 · Gradient Descent: The Ultimate Optimizer (NeurIPS 2022 · 5 citations)
- 2023 · DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method (NeurIPS 2023 · 2 citations)
- 2024 · The Road Less Scheduled (NeurIPS 2024 · 16 citations)
- 2025 · Momentum-SAM: Sharpness Aware Minimization without Computational Overhead (NeurIPS 2025 · 2 citations)
- 2026 · RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization (ICML 2026 · 0 citations)
Earliest papers here
- Expected Tensor Decomposition with Stochastic Gradient Descent (AAAI 2016 · 41 citations)
- Scalable Adaptive Stochastic Optimization Using Random Projections (NeurIPS 2016 · 7 citations)
- Factorization Ranking Model for Move Prediction in the Game of Go (AAAI 2016 · 6 citations)
What has been tried
51 papers here report 505 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 13%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use momentum β=0.9 instead of β=0.98 | NanoGPT on OpenWebText · loss | Hurt | The Road Less Scheduled (NeurIPS 2024 · 16 citations) | |
| add adaptive parameter-scaled perturbations | CIFAR100 · test accuracy | 83.21 -> 84.15; 79.11 -> 79.89; 82.65 -> 83.48 | Mixed | Momentum-SAM: Sharpness Aware Minimization without Computational Overhead (NeurIPS 2025 · 2 citations) |
| add gradient compensation to PID-controlled SGD | CIFAR-100 · accuracy | 72.5% -> 74.7% | Helped | A PID Controller Approach for Adaptive Probability-dependent Gradient Decay in Model Calibration (NeurIPS 2024 · 1 citation) |
| Replace online gradient descent with AdaGrad for learning the preconditioner | Optimization problems · convergence speed | Helped | Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent (ICML 2025 · 0 citations) | |
| Use Subspace-Momentum rank 32 instead of rank 64 | LLaMA 7B SFT, UltraFeedback · last validation perplexity | 2.584 -> 2.580 | Helped | Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · 0 citations) |
| Cache eigendecomposition results for 100 steps rather than recomputing more frequently | LLM-Init · time-to-Adam | 3.34 -> 0.74 | Helped | A Stable Whitening Optimizer for Efficient Neural Network Training (NeurIPS 2025 · 0 citations) |
| Use diagonal preconditioning (DASGO) instead of full-matrix ASGO | GPT-2 pretraining · validation loss | 3.342 -> 3.509 | Hurt | ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · 0 citations) |
| Use discounting (MagL-D) instead of the undiscounted MagL | Image classification (time-varying corruption) · LCE_100 | 0.09 -> 0.08 | Helped | Discounted Adaptive Online Learning: Towards Better Regularization (ICML 2024 · 0 citations) |
| Vary η across the reported settings | CIFAR10 · top-1 accuracy | Helped | Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · 0 citations) | |
| enable early stopping for the PG subproblem solver | CIFAR10 validation · accuracy | 92.7 ± 0.2\% -> 92.9 ± 0.2\% | No clear effect | Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 3 of 17 papers
- CIFAR10 · 2 of 17 papers
- CIFAR100 · 2 of 17 papers
- OpenWebText · 2 of 17 papers
- ImageNet-1k · 2 of 17 papers
- mnist · 2 of 17 papers
- FineWeb · 2 of 17 papers
Baselines
- Adam · 7 of 17 papers
- AdamW · 5 of 17 papers
- Muon · 4 of 17 papers
- Shampoo · 3 of 17 papers
- AdaGrad · 2 of 17 papers
- SGD · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 42 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- Amazon · 3 papers
- Princeton University · 3 papers
- University of Texas, Austin · 2 papers
- University of Texas at Austin · 2 papers
- EPFL · 2 papers
- Toyota Technological Institute at Chicago · 2 papers
- Samsung · 2 papers
- University of British Columbia · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Adam convergence and adaptive learning rates (78 papers)
- Memory-efficient LLM optimizers (79 papers)
- SGD momentum and step-size schedules (72 papers)
Elsewhere in the topic
- Stochastic gradient MCMC (142 papers)
- Surrogate losses and robust solvers (119 papers)
- Differentiating through embedded optimization (117 papers)
- Constrained and non-backprop training (97 papers)
- Natural gradient and Fisher approximations (77 papers)
- Randomized gradient estimation (77 papers)
- Stein variational gradient descent (47 papers)
- Gradient-based meta-learning theory (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).