Research map / Optimization / Implicit bias and generalization
The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a nonlinear activation instead of linear | single-hidden-layer network Fisher · theory, Gaussian data | conditioning m2/m1^2 and spectrum degeneracy: lambda_min 0 -> 0.086 (erf1) | Helped |
| use fopt activation instead of erf1 | Fisher spectrum · theory | lambda_max: 7.51 -> 2.247 | Helped |
| tune activation parameter alpha (srelu_alpha, erf_alpha) toward lower m2/m1^2 | student-teacher net, width 2^7 · single-step loss reduction | single-step loss decrease | Helped |
| use small-width simulation to infer large-width spectrum | empirical Fisher spectra · erf1 widths 16-128 | agreement with infinite-width theory | Mixed |
Closest papers by meaning
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (ICML 2019 · similarity 0.90)
- Geometry of Neural Network Loss Surfaces via Random Matrix Theory (ICML 2017 · similarity 0.89)
- Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.86)
- Trade-Offs of Diagonal Fisher Information Matrix Estimators (NeurIPS 2024 · similarity 0.86)
- Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.