Research map / Learning theory and algorithms / Matrix and tensor methods
Nonlinear random matrix theory for deep learning
Jeffrey Pennington, Pratik Worah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set zeta = E[f'(z)]^2 = 0 (activation with zero squared mean derivative, e.g., f1-f4 in Fig. 3) | Gram matrix Y^T Y of single-layer nonlinear random features · Analytic limit / data covariance | limiting spectral distribution | No clear effect |
| Choose activation with zeta near 0 (small nonlinearity mismatch) instead of zeta near eta | Random feature ridge regression memorization · Memorization task, theory vs simulation | training loss E_train | Improved |
| Propagate data covariance through deep network with activation having zeta=0 (vs zeta != 0) (ablation) | Data covariance eigenvalue distribution at layer 10 · Piecewise linear relu-variant activations, width sweep | distance d between layer-1 and layer-10 eigenvalue distributions | Improved |
| Replace linear activation with nonlinear activation (eta=zeta not satisfied) (ablation) | Random feature memorization capacity · Simulated random feature networks, beta sweep | training loss E_train | Improved |
Closest papers by meaning
- Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping (JMLR 2023 · similarity 0.88)
- Geometry of Neural Network Loss Surfaces via Random Matrix Theory (ICML 2017 · similarity 0.88)
- The Dynamics of Learning: A Random Matrix Approach (ICML 2018 · similarity 0.88)
- Analysis of one-hidden-layer neural networks via the resolvent method (NeurIPS 2021 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.85)
- Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
