Research map / Architectures and efficiency / RNNs and normalization
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that an MLP with linear activations, batch normalization, and Haar-orthogonal weights provably has bounded gradients at any depth and exponentially decaying isometry gap, and proposes gain-based activation shaping to extend this to tanh and sin.
Problem
Batch normalization avoids rank collapse in deep networks but causes exponentially exploding gradients with depth at initialization, limiting trainable depth.
Approach
Use orthogonal (Haar) random weights instead of Gaussian in a BN-MLP with linear activations, where the isometry gap (a Lyapunov function) decays exponentially in depth via Weingarten calculus, so log gradient norms stay bounded non-asymptotically; extend to nonlinear activations via per-layer gain shaping.
Key results
- Isometry gap of representations decays at an exponential rate in depth for orthogonal-weight BN-MLPs, matching the theoretical prediction on random data.
- For orthogonal weights with BN, log gradient norm of the first layer stays bounded across depths, whereas Gaussian weights with BN show exponential gradient growth.
- Vanilla SGD on CIFAR10 with linear MLPs (width 100, batch 100, lr 0.001) shows an almost depth-independent convergence rate; shaped tanh and sin achieve similar depth-independent training accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use orthogonal instead of Gaussian weight initialization | MLP with batch normalization · CIFAR10, gradient norm at initialization | gradient norm growth with depth | Helped |
| use orthogonal weights and batch normalization with non-degenerate inputs | linear-activation MLP · arbitrary depth | gradient norm | Helped |
| use orthogonal weights with batch normalization | MLP · arbitrary depth, full-rank input batches | isometry gap | Helped |
| use degenerate rather than non-degenerate input batches | MLP with batch normalization · gradient norm at initialization | gradient norm growth with depth | Hurt |
| shape tanh or sin activations toward identity using per-layer pre-activation gain | MLP with batch normalization and orthogonal weights · CIFAR10, gradient norm at initialization | gradient norm growth with depth | Helped |
| use activation shaping with nonlinear activations | MLP with batch normalization · CIFAR10 training | training accuracy across depths | Helped |
| use identity rather than shaped sin or tanh activations (ablation) | MLP with batch normalization · CIFAR10 training | training accuracy | Hurt |
| use standard batch normalization instead of the paper's modified batch-normalization operator (ablation) | constructed MLP · signal propagation and gradient explosion | gradient explosion and signal propagation | No clear effect |
| add nonlinear activations without shaping (ablation) | MLP with batch normalization · deep networks | gradient norm growth with depth | Hurt |
Surprising findings
- Middle layers of very deep (1000-layer) MLPs remain nearly orthogonal during SGD training, suggesting an implicit orthogonality bias of SGD.
- The identity activation trains much worse than shaped tanh and sin, indicating the nonlinear networks are not operating in the linear regime.
Limitations
- Theory holds only for linear activations and a simplified BN module (no mean reduction, no 1/n factor).
- Requires input batches to be linearly independent (non-degenerate); degenerate batches yield unbounded bounds.
- Activation shaping for nonlinear activations is empirical, without theoretical guarantees.
Experimental setup
Datasets: CIFAR10 CIFAR100 MNIST FashionMNIST
Benchmarks: CIFAR10 CIFAR100 MNIST FashionMNIST
Metrics: isometry gap log gradient norm training accuracy
Baselines: Gaussian weights with batch normalization identity activation shaped tanh shaped sin
Builds on: batch normalization dynamical isometry deep kernel shaping mean-field theory of batch normalization
Closest papers by meaning
- A Mean Field Theory of Batch Normalization (ICLR 2019 · similarity 0.94)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.91)
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (ICML 2016 · similarity 0.91)
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (NeurIPS 2020 · similarity 0.90)
- On the impact of activation and normalization in obtaining isometric embeddings at initialization (NeurIPS 2023 · similarity 0.89)
- Batch normalization is sufficient for universal function approximation in CNNs (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.