Research map / Architectures and efficiency / RNNs and normalization
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Subtract the last-layer output mean | Wide non-centered DNNs · Empirical FIM with T=O(M) | largest eigenvalue: Without mean subtraction: λ_max∼α ((T-1)/T κ_2 + κ_1/T)M; with mean subtraction: ρ α (κ_1 - κ_2) + c_1 M^{1-2q*} ≤ λ_max | Helped |
| Add variance normalization to last-layer mean subtraction (ablation) | Wide non-centered DNNs · Empirical FIM with T=O(M) | largest eigenvalue: The variance normalization changes the eigenvalue statistics by constant factors. | No clear effect |
| Subtract the last-layer output mean (ablation) | Wide non-centered DNNs · Empirical FIM with T≥2 and T=O(1) | largest eigenvalue: λ_max ∼ α (κ_1-κ_2)/T M | No clear effect |
| Normalize the middle layers while leaving the last layer unnormalized | Wide DNNs with non-negative activations · Empirical FIM | largest eigenvalue: λ_max ≥ α_{L-1} ((T-1)/T q̂^{L-1}_{st,BN} + q̂^{L-1}_{t,BN}/T) M | No clear effect |
| Apply layer normalization across units in each sample | Wide non-centered DNNs · Empirical FIM, C>2 | largest eigenvalue: αs/((C-2)η_1κ'_1) M ≤ λ_max ≤ α√s M | No clear effect |
| Subtract the last-layer output mean | Wide ReLU DNNs trained by gradient descent · Training loss across network widths and learning rates | convergence learning-rate threshold: η = 2/(ρ α (κ_1-κ_2)) | Helped |
Closest papers by meaning
- Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction (NeurIPS 2022 · similarity 0.90)
- Batch normalization provably avoids ranks collapse for randomly initialised deep networks (NeurIPS 2020 · similarity 0.89)
- On the impact of activation and normalization in obtaining isometric embeddings at initialization (NeurIPS 2023 · similarity 0.89)
- Characterizing Well-Behaved vs. Pathological Deep Neural Networks (ICML 2019 · similarity 0.88)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.88)
- On the Importance of Gaussianizing Representations (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.