Research map / Optimization / Optimizers
Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the exact Fisher information with a positive-definite layer-wise approximation | NGD · Infinite-width fully connected DNNs (training samples) | convergence rate: f_t = y+(1-c)^t (f_0-y); c=1 | Helped |
| Use a block-diagonal Fisher approximation instead of the exact Fisher | NGD · Infinite-width fully connected DNNs (training samples) | convergence rate: \alpha=L | Helped |
| Use a block tri-diagonal Fisher approximation | NGD · Infinite-width fully connected DNNs (training samples) | convergence: L=3s or 3s+1; L=3s+2 | Mixed |
| Use a K-FAC approximation | NGD · Infinite-width fully connected DNNs (training samples; C=1, M_0 ≥ N) | convergence rate: \bar{\Theta}=NLI; \eta_{\text{opt}}=1/(NL) | Helped |
| Use a unit-wise Fisher approximation under the gradient independence assumption | NGD · Infinite-width fully connected DNNs (training samples; C=1) | convergence rate: \alpha=\gamma M(L-1) | Helped |
| Use an entry-wise diagonal Fisher approximation instead of a layer-wise or unit-wise approximation | NGD · Infinite-width fully connected DNNs (training samples) | condition number of the function-space coefficient matrix | Mixed |
| Use the block tri-diagonal approximation at depth L=3s+2 (ablation) | Block tri-diagonal NGD · Infinite-width fully connected DNNs (training samples) | convergence: L=3s+2 | Hurt |
| Use K-FAC without input preprocessing when M_0 < N (ablation) | K-FAC · Infinite-width fully connected DNNs (training samples; M_0 < N) | isotropy: \frac{1}{N}\bar{\Theta} = (L-1)I + (I \odot X (X^\top X)^{-1} X^\top) | Hurt |
| Apply the Forster transformation to inputs for K-FAC when M_0 < N (ablation) | K-FAC · Infinite-width fully connected DNNs (training samples; M_0 < N) | isotropy: \bar{\Theta} = \alpha I, \ \alpha = N(L-1) + M_0 | Helped |
Closest papers by meaning
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.91)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.90)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.87)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.87)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.87)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.