Research map / Architectures and efficiency / RNNs and normalization
Traditional and Heavy Tailed Self Regularization in Neural Network Models
Michael Mahoney, Charles Martin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease batch size from 500 to 2 (ablation) | MiniAlexNet · CIFAR10 | test accuracy | Mixed |
| decrease batch size (ablation) | MiniAlexNet · CIFAR10 | MP Soft Rank | Improved |
| decrease batch size (ablation) | MiniAlexNet · CIFAR10 | Stable Rank | Improved |
| decrease batch size (ablation) | MiniAlexNet · CIFAR10 | training accuracy | Mixed |
| add explicit regularization, such as L2 weight regularization or Dropout (ablation) | MiniAlexNet · CIFAR10 | complexity metrics (entropy and Stable Rank) | Improved |
| train the model | MiniAlexNet · CIFAR10 | largest eigenvalue λmax: roughly 3.0 to roughly 4.0 | Mixed |
Closest papers by meaning
- Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (JMLR 2021 · similarity 0.90)
- Models of Heavy-Tailed Mechanistic Universality (ICML 2025 · similarity 0.85)
- The Underlying Universal Statistical Structure of Natural Datasets (ICML 2025 · similarity 0.83)
- Weight matrices compression based on PDB model in deep neural networks (ICML 2025 · similarity 0.81)
- Implicit Regularization Paths of Weighted Neural Representations (NeurIPS 2024 · similarity 0.81)
- Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
