Research map / Architectures and efficiency / RNNs and normalization
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
Charles H. Martin, Michael W. Mahoney
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train MLP3 with backprop (versus random initialization) | MLP3 FC2 · CIFAR10 | matrix entropy: 0.92 -> just below 0.91 | Improved |
| train MLP3 with backprop (versus random initialization) | MLP3 FC2 · CIFAR10 | stable rank: decreases by approximately a factor of two | Improved |
| training MLP3 FC2: spectral norm squared | MLP3 FC2 · CIFAR10 | squared spectral norm: lambda_max 3.2 -> 7.2 | Improved |
| older small model (LeNet5) versus large modern models (AlexNet) ESD fit to MP | ESD analysis · Pretrained DNN layers | fit to Marchenko-Pastur: LeNet5 lambda+ ~ 3.5 well fit; AlexNet alpha 2.25-3.02 heavy-tailed | Hurt |
Closest papers by meaning
- Traditional and Heavy Tailed Self Regularization in Neural Network Models (ICML 2019 · similarity 0.90)
- Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks (NeurIPS 2022 · similarity 0.89)
- Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping (JMLR 2023 · similarity 0.89)
- Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization (ICML 2026 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Implicit Regularization in Deep Learning May Not Be Explainable by Norms (NeurIPS 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
