Research map / Optimization / Optimizers
An Improved Empirical Fisher Approximation for Natural Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes improved empirical Fisher (iEF), which rescales per-sample gradients by logit-gradient norms to fix EF's inversely-scaled projection issue, and an efficient framework to evaluate approximate NG update quality, showing iEF beats EF and sampled Fisher on approximation quality and PEFT optimisation.
Problem
Empirical Fisher (EF) is a widely used but poor approximation to the exact Fisher / natural gradient, due to an inversely-scaled projection issue; existing evaluations of approximation quality need explicit exact NG computation and do not scale to large models.
Approach
Replace the EF scaling vector of ones with per-sample squared logit-gradient norms (iEF), which induces per-sample loss reductions tied to convergence level and is equivalent to Gauss-Newton; also define an indicator gamma computed with Fisher-vector products to score approximate NG updates at scale.
Key results
- On GLUE with T5 Prompt Tuning, iEF averages 79.3 vs 77.1 for Adafactor, 69.7 for SF, and 48.1 for EF.
- On CIFAR100 ViT+LoRA, iEF averages 94.3 test vs 93.9 for AdamW, 92.8 for SF and 31.0 for EF.
- Under the gamma indicator, approximation quality follows iEF > SF > EF across most training stages and tasks, and iEF is robust to near-zero damping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Adafactor with iEF | T5-base with Prompt Tuning · GLUE + T5 + Prompt Tuning | average test performance: 77.1 -> 79.3 | Helped |
| Replace AdamW with iEF | T5-base with LoRA · GLUE + T5 + LoRA | average test performance: 80.1 -> 79.3 | Hurt |
| Replace AdamW with iEF | ViT with LoRA · CIFAR100 + ViT + LoRA | average test performance: 93.9 -> 94.3 | Helped |
| Use iEF instead of EF | Approximate NG updates across the evaluated setups · 15 experimental setups | approximation quality to exact NG updates | Helped |
| Use iEF instead of SGD | Approximate NG updates across the evaluated setups · 15 experimental setups | approximation quality to exact NG updates | Helped |
| Use near-zero damping with iEF rather than relying on damping tuning | iEF approximation · Across tasks and training stages | approximation quality to exact NG updates | Helped |
| Use damping with EF or SF | EF and SF approximations · Across tasks and training stages | approximation quality to exact NG updates | Helped |
| Use EF as the update method rather than SGD | Approximate NG updates · Most training stages and tasks | approximation quality to exact NG updates | Hurt |
| Use iEF rather than the corresponding Adafactor or AdamW baseline | Optimisation of the evaluated tasks · 15 experimental setups | test performance | Mixed |
Surprising findings
- EF performs worse than SGD as an approximation to the natural gradient for most training stages and tasks.
- EF updates become larger as training progresses and samples converge, an inversely-scaled effect that also hurts SF.
Limitations
- Exact iEF/EF memory scales as O(MP), restricting use to small trainable parameter counts such as PEFT.
- LoRA results lag behind AdamW, attributed to missing integration of regularisation and momentum with iEF.
- Convergence analysis is continuous-time and non-stochastic, and the strongly convex result does not cover cross-entropy.
Experimental setup
Datasets: GLUE CIFAR100 CIFAR10
Benchmarks: GLUE CIFAR100 CIFAR10
Metrics: test performance final training loss approximation quality indicator gamma
Baselines: AdamW Adafactor SGD EF SF
Builds on: natural gradient descent Gauss-Newton method K-FAC Hessian-free optimisation WoodFisher
Closest papers by meaning
- Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks (NeurIPS 2020 · similarity 0.91)
- Limitations of the empirical Fisher approximation for natural gradient descent (NeurIPS 2019 · similarity 0.90)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.89)
- Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training (AAAI 2026 · similarity 0.88)
- Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis (NeurIPS 2018 · similarity 0.87)
- Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.