Research map / Optimization / Implicit bias and generalization
A Mean Field Theory of Batch Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use batch normalization in a deep network without skip connections | fully connected networks at initialization · wide networks with a BSB1 fixed point | gradient norm explosion rate | Hurt |
| tune the activation closer to the linear regime (ablation) | deep batch-normalized networks · MNIST | maximum trainable depth | Helped |
| increase ε (ablation) | batch-normalized networks · MNIST | maximum trainable depth | Helped |
| use the identity activation instead of a nonlinear activation (ablation) | batch-normalized networks · wide networks with a BSB1 fixed point | gradient norm explosion rate: B-2/B-3; ReLU: π/(π-1) ≈ 1.467 as B → ∞ | Helped |
| increase ε (ablation) | VGG19 with batch normalization · CIFAR100 | training accuracy | Helped |
| initialize β above zero (ablation) | VGG19 with batch normalization · CIFAR100 | training accuracy | Helped |
| train for one optimization step | batch-normalized networks · gradient magnitudes across depth | gradient dynamic range | Helped |
| increase network depth | batch-normalized networks processing similar batches · cross-batch correlation | correlation | Hurt |
Closest papers by meaning
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.94)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.91)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.90)
- Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (AAAI 2019 · similarity 0.90)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.90)
- How Does Batch Normalization Help Optimization? (NeurIPS 2018 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.