Research map / Optimization / Implicit bias and generalization
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Weight Normalization with a suitable initialization magnitude α* instead of un-normalized training | Two-layer ReLU network · General regression data | finite-step convergence rate: (1-\eta_{\alpha^*}\lambda_{\min}(\bm{\Lambda}(s))) \le (1 - \eta \lambda_{\min}(\mathbf{H}(s))) | Helped |
| Use the G-dominated Weight Normalization regime with α = \Omega(\sqrt{n/\mu_0}) | Two-layer ReLU network · General regression data | required width: n^6 to n^2; m = \Omega \bigg( \frac{n^2 \log{(n/\delta)}}{\mu_0^2} \bigg) | Helped |
| Reduce α in the V-dominated regime | Weight Normalization · General regression data, α < 1 | gradient-flow convergence rate: \lambda_0/\alpha^2 | Helped |
| Reduce α in finite-step V-dominated gradient descent | Weight Normalization · General regression data, α < 1 | allowed step size: \eta = O(\alpha^2 /\|\mathbf{V}^{\infty}\|_2) | Hurt |
| Use α ≠ 1 to modify the Weight Normalization NTK | Weight Normalization · General regression data | convergence rate | Helped |
| Set α = 1 | Weight Normalization · Initialization | initial NTK: \mathbf{V}(0) + \mathbf{G}(0) = \mathbf{H}(0) | No clear effect |
| Use Weight Normalization rather than un-normalized training | Two-layer ReLU network · General regression data | required width dependence on failure probability: \log(1/ \delta) versus polynomial dependence of order \delta^3 | Helped |
| Use the paper's refined over-parametrization analysis | Two-layer ReLU network convergence bounds · General regression data | required width dependence on sample size: n^6 to n^4 \log(n) | Helped |
Closest papers by meaning
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.93)
- Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs (ICML 2021 · similarity 0.91)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.89)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.88)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)
- Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.