Research map / Learning theory and algorithms / Matrix and tensor methods
Trade-Offs of Diagonal Fisher Information Matrix Estimators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes variances of the two standard random estimators of the diagonal Fisher information matrix of neural networks, derives closed-form and computable element-wise bounds, and examines regression/classification trade-offs empirically on MNIST.
Problem
Practitioners estimate diagonal entries of the Fisher information matrix (FIM) of neural networks with random estimators, but the variances (which govern accuracy and sample complexity) of the two common estimators are hard to compute for deep networks because of the huge parameter dimension.
Approach
Derives element-wise (per-parameter) closed-form variances and bounds for the two diagonal FIM estimators using variational eigenvalue characterizations, expressing them as products of NN derivatives and exponential-family sufficient-statistic moments, so they can be computed via automatic differentiation; extends the analysis to joint FIM via the law of total variance and to the empirical Fisher.
Key results
- For classification on MNIST MLPs, the variance of the second estimator is zero in the last layer, so it is preferable there, while the first estimator has the largest variance in that layer.
- Upper bounds are valid across architectures and activations and are roughly 10 to 50 times larger than the true variances, serving as order-of-magnitude hints.
- For regression with an isotropic Gaussian, the eigenvalue bounds are exact, making the conditional FIM and variance bounds equalities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FIM estimator 1 instead of FIM estimator 2 | Diagonal FIM estimation in the first three layers of the MNIST MLP · MNIST test set, first three MLP layers | conditional estimator variance | Helped |
| Use FIM estimator 2 instead of FIM estimator 1 | Diagonal FIM estimation in the last layer of the MNIST MLP · MNIST test set, last MLP layer | conditional estimator variance: zero | Helped |
| Use FIM estimator 2 rather than FIM estimator 1 | Diagonal FIM estimation for parameters in the last network layer · Neural networks, last layer | estimator variance: zero variance | Helped |
| Use FIM estimator 2 rather than FIM estimator 1 | Diagonal FIM estimation for weights in the final linear output layer · Neural networks, final layer | conditional estimator variance: 0 | Helped |
| Use log-sigmoid activations | MLP FIM estimators during training · MNIST | conditional estimator variance | Helped |
| Use estimator 2 instead of estimator 1 for regression | Diagonal FIM estimation · Regression | estimator variance | Helped |
| Use estimator 1 instead of estimator 2 for ReLU networks | Diagonal FIM estimation · ReLU networks | applicability | Mixed |
Surprising findings
- The second (Hessian-based) estimator has zero variance in the last layer, while the first estimator is most variable there.
- With log-sigmoid activations the variances vanish quickly after random initialization, suggesting training reaches near-linear regions.
Limitations
- Hessian-based estimator requires costly second derivatives and memory scaling quadratically for non-linear activations, requiring approximations.
- Lower bounds for the tensor eigenvalue terms are often trivial; tensor eigenvalues are expensive for large output dimension T.
- Empirical study is limited to small MLPs on MNIST.
Experimental setup
Datasets: MNIST synthetic linear-Gaussian/Bernoulli toy data
Benchmarks: MNIST
Metrics: variance of diagonal FIM estimators log-ratio of upper bound to variance test loss in natural gradient toy example
Baselines: empirical Fisher generalized Gauss-Newton
Builds on: natural gradient Adam Monte Carlo information geometry
Closest papers by meaning
- On the Variance of the Fisher Information for Deep Learning (NeurIPS 2021 · similarity 0.90)
- Relative Fisher Information and Natural Gradient for Learning Large Modular Models (ICML 2017 · similarity 0.88)
- Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator (ICML 2025 · similarity 0.86)
- Fisher information and the geometry of memorization in neural networks (NeurIPS 2026 · similarity 0.84)
- Sketched Lanczos uncertainty score: a low-memory summary of the Fisher information (NeurIPS 2024 · similarity 0.84)
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.