Research map / Optimization / Optimizers
Natural gradient and Fisher approximations
Develops and critiques Kronecker-factored and structured approximations of natural gradient and second-order updates.
77 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Optimizers, keyword cluster “variational · sampling · inference”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.4% to 6.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML 2016 · 82 citations)
- 2017 · Distributed Second-Order Optimization using Kronecker-Factored Approximations (ICLR 2017 · 66 citations)
- 2018 · Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis (NeurIPS 2018 · 68 citations)
- 2019 · A Variational Inequality Perspective on Generative Adversarial Networks (ICLR 2019 · 157 citations)
- 2020 · Practical Quasi-Newton Methods for Training Deep Neural Networks (NeurIPS 2020 · 46 citations)
- 2021 · Efficient Neural Network Training via Forward and Backward Propagation Sparsification (NeurIPS 2021 · 15 citations)
- 2022 · Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network (NeurIPS 2022 · 8 citations)
- 2023 · Learning GFlowNets From Partial Episodes For Improved Convergence And Stability (ICML 2023 · 5 citations)
- 2024 · An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · 2 citations)
- 2025 · Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization (NeurIPS 2025 · 2 citations)
- 2026 · Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation (ICML 2026 · 0 citations)
Earliest papers here
- Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML 2016 · 82 citations)
- Provable Efficient Online Matrix Completion via Non-convex Stochastic Gradient Descent (NeurIPS 2016 · 41 citations)
- A Kronecker-factored approximate Fisher matrix for convolution layers (ICML 2016 · 31 citations)
What has been tried
62 papers here report 525 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 14%, hurt 18%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use 10% sketch-size Nyström randomization rather than exact Woodbury computation | 5D Poisson, large batches · training speed | 2× | Mixed | Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization (NeurIPS 2025 · 2 citations) |
| use exact Hessian diagonals at the last layer instead of the BL89 approximation | MNIST classification, Hessian-diagonal approximation · L1 approximation error | Helped | Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning (ICML 2024 · 1 citation) | |
| Add sampling approximation of the Fisher information matrix | ImageNet-1K · total time to 75.9% test accuracy | 7.43h -> 6.46h | Helped | A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · 1 citation) |
| Add dynamic programming to backpropagation | MC2021 competition track 2, Prob semiring · Time (ms) | 19.8 ± 0.2 -> 7.8 ± 0.1 | Helped | The Gradient of Algebraic Model Counting (AAAI 2025 · 0 citations) |
| add gradient momentum in the perturbation step | CIFAR-10 · accuracy | 92.98[0.05] -> 92.36[0.23] | Hurt | Sassha: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation (ICML 2025 · 0 citations) |
| Remove Squisher rescaling and use the Fisher-default lambda | Split CIFAR-100 task-incremental · accuracy | 75.30 -> 61.53 | Hurt | Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator (ICML 2025 · 0 citations) |
| restrict posterior sampling to batch-normalization layers rather than perturbing all parameters | CIFAR-10, corruption 0.0–0.5 · accuracy | 0.8654 ± 0.0094 -> 0.8757 ± 0.0029; 0.8479 ± 0.0081 -> 0.8544 ± 0.0034; 0.8079 ± 0.0084 -> 0.8120 ± 0.0058; 0.7288 ± 0.0 | Helped | Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes (NeurIPS 2025 · 0 citations) |
| increase batch size from 64 to 128 | CIFAR-100, ResNet-50 · Top-1 error | 20.27 -> 20.38 | Hurt | KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products (NeurIPS 2025 · 0 citations) |
| Use Tanh instead of ReLU augmentation | Debutanizer and SRU · R² | 0.5654 -> 0.7216; 0.8115 -> 0.8026 | Mixed | Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient (NeurIPS 2024 · 0 citations) |
| use single-pass rather than two-pass sketching | Heat equation, A1 (261 parameters) · relative L2 error (median) | 1.49E-07 -> 3.62E-07 | Hurt | Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks (ICML 2025) |
| Use diagonal instead of dense Kronecker factors | ImageWoof-10 (GCViT) · test error | No clear effect | Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 16 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 4 of 16 papers
- CIFAR-100 · 4 of 16 papers
- ImageNet · 2 of 16 papers
- Wikitext-2 · 2 of 16 papers
- MNIST · 2 of 16 papers
Baselines
- Adam · 5 of 16 papers
- Shampoo · 4 of 16 papers
- K-FAC · 4 of 16 papers
- SGD · 3 of 16 papers
- AdamW · 3 of 16 papers
- AdaHessian · 2 of 16 papers
- ENGD · 2 of 16 papers
- gradient descent · 2 of 16 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 37 papers; from OpenAlex and the papers' affiliations).
- Vector Institute · 4 papers
- KU Leuven · 3 papers
- University of Cambridge · 3 papers
- MedaiTek Research · 2 papers
- University of Toronto · 2 papers
- Université de Montréal · 2 papers
- University of Montreal · 2 papers
- University of Texas, Austin · 2 papers
- University of Texas at Austin · 2 papers
- Google · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Stochastic gradient MCMC (142 papers)
- Stein variational gradient descent (47 papers)
- Zeroth-order training methods (23 papers)
Elsewhere in the topic
- Surrogate losses and robust solvers (119 papers)
- Differentiating through embedded optimization (117 papers)
- Constrained and non-backprop training (97 papers)
- Memory-efficient LLM optimizers (79 papers)
- Adam convergence and adaptive learning rates (78 papers)
- Randomized gradient estimation (77 papers)
- SGD momentum and step-size schedules (72 papers)
- Second-order and structured preconditioning (70 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).