Research map / Optimization / Implicit bias and generalization
A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply batch normalization with a separate learning rate for the weights | OLS gradient descent · OLS | convergence stability | Helped |
| Use batch normalization, introducing a rescaling parameter and over-parameterization | OLS gradient descent · OLS | asymptotic convergence rate | Helped |
| Use batch normalization instead of ordinary gradient descent | OLS gradient descent · OLS | sensitivity to learning rate | Helped |
| Separate the learning rate for batch-normalization parameters from the weight learning rate (ablation) | BNGD · MNIST, Fashion MNIST, and CIFAR-10 classification experiments | stability at large learning rates | Helped |
| Use shared learning rates for weights and batch-normalization parameters rather than separate rates (ablation) | BNGD · MNIST, Fashion MNIST, and CIFAR-10 classification experiments | stability at large learning rates | Hurt |
| Vary the weight learning rate while retaining batch normalization (ablation) | BNGD · OLS | sensitivity to learning rate | Helped |
| Use batch normalization in deeper networks rather than a one-layer network (ablation) | BNGD · MNIST, Fashion MNIST, and CIFAR-10 classification experiments | convergence acceleration | Mixed |
| Use a small initial value for the batch-normalization scale parameter (ablation) | BNGD · OLS | learning-rate robustness | Helped |
| Use a small learning rate for the batch-normalization scale parameter (ablation) | BNGD · OLS | optimization performance | Hurt |
| Increase hidden-layer width (ablation) | Stochastic BNGD · MNIST | learning-rate insensitivity interval | Helped |
Closest papers by meaning
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (ICLR 2019 · similarity 0.90)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.90)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.90)
- On the Training Instability of Shuffling SGD with Batch Normalization (ICML 2023 · similarity 0.88)
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization (ICLR 2020 · similarity 0.88)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.