Research map / Optimization / Optimizers
Limitations of the empirical Fisher approximation for natural gradient descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use empirical Fisher instead of Fisher as the curvature preconditioner | Natural gradient descent · Linear regression | optimization performance | Hurt |
| Use empirical Fisher instead of Fisher as the curvature preconditioner | Natural gradient descent · Linear classification | optimization performance | Hurt |
| Precondition with the empirical Fisher far from an optimum | Gradient descent · Optimization far from an optimum | update direction agreement with Fisher-preconditioned gradient | Hurt |
| Precondition with the inverse empirical Fisher far from an optimum | Gradient descent · Optimization far from an optimum | update scaling | Hurt |
| Use empirical Fisher as a second-order approximation when individual training gradients become small | Empirical Fisher · Overparameterized models at a training minimum | curvature approximation quality | Hurt |
| Use empirical Fisher as an approximation to Fisher near a minimum without realizability and sufficient data | Empirical Fisher · Misspecified or data-limited models | Fisher approximation quality | Hurt |
| Interpret empirical Fisher as adaptation to gradient noise rather than curvature | Empirical Fisher-based methods · Stochastic optimization | gradient-noise adaptation | Helped |
Closest papers by meaning
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.90)
- Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator (ICML 2025 · similarity 0.88)
- Energetic Natural Gradient Descent (ICML 2016 · similarity 0.87)
- Relative Fisher Information and Natural Gradient for Learning Large Modular Models (ICML 2017 · similarity 0.87)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.85)
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.