Research map / Architectures and efficiency / RNNs and normalization
Batch Normalization Is Blind to the First and Second Derivatives of the Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves via Taylor expansion of the loss that batch normalization's standardization phase blocks the influence of first-order and diagonal second-order loss terms on parameters before BN, and validates this with synthetic MLP, VGG, RealNVP and feature-visualization experiments.
Problem
Batch normalization is analyzed for how it affects back-propagation of loss derivatives; the paper asks which loss-derivative terms (first and second order) BN blocks from reaching layers before it.
Approach
Decomposes the batch loss via Taylor expansion w.r.t. the standardized features and proves that the derivatives of the batch mean and standard deviation in the standardization phase cancel the gradient and diagonal-Hessian terms, so these cannot influence earlier layers; off-diagonal Hessian effects are only partly blocked.
Key results
- In MLPs with BN, relative gradient differences for first and second orders were near zero (about 1.8e-7 and 2.9e-7) versus about 1.0 without BN.
- Blocked first-order and diagonal-Hessian terms accounted for 79.5%, 66.2%, and 37.8% of loss value for VGG-16, VGG-11, and AlexNet respectively (CIFAR-10).
- Gradient norm of the linear off-diagonal term L_linear with respect to x_d is about 1e-11 to 1e-10 with BN versus around 1e-3 without BN on MNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply batch normalization | MLP · Synthetic loss functions | ∆grad1: 1.04±0.51 -> 1.79e-7±2.1e-7 | Hurt |
| apply batch normalization | MLP · Synthetic loss functions | ∆grad2: 1.07±0.57 -> 2.85e-7±3.9e-8 | Hurt |
| apply batch normalization | VGG-11 · CIFAR-10 image classification | ∆gradfirst: 3.58±0.36 -> 1.37e-8±2.72e-8 | Hurt |
| apply batch normalization | VGG-11 · CIFAR-10 image classification | ∆gradsecond,diag: 2.22±0.26 -> 2.72e-8±5.41e-8 | Hurt |
| apply batch normalization | VGG-11 · CIFAR-10 image classification | ∆gradsecond,off: 100.70±11.78 -> 1019.49±116.45 | Mixed |
| apply batch normalization | VGG-16, VGG-11, and AlexNet · CIFAR-10 image classification | pblocked: 0.795 ± 0.397; 0.662 ± 0.303; 0.378 ± 0.200 | Hurt |
| replace batch normalization with layer normalization (ablation) | RealNVP · MNIST image generation | log-likelihood separation of real and interpolated images | Helped |
| replace batch normalization with layer normalization (ablation) | DNN classifier · CIFAR-10 with category-specific mini-batches | feature clustering | Helped |
| train with samples of different labels in each mini-batch (ablation) | DNN with BN · CIFAR-10 classification | feature representation | Helped |
| remove the BN layer (ablation) | AlexNet-1/2/3 and LeNet-1/2/3 · MNIST | ∥∂Llinear_d/∂xd∥ | Hurt |
| remove the BN layer (ablation) | VGG-11 · CIFAR-10 image classification | ∆gradfirst and ∆gradsecond,diag | Helped |
Surprising findings
- Blocking by BN is claimed to hold only in the training phase, not at test time.
- In mini-batches with differing sample losses (general case), the blocked terms have less effect on feature quality for classification, since residual and off-diagonal terms suffice.
Limitations
- The blocking problem is not fully equivalent to significant damage on benchmark tasks.
- The Taylor residual term can be large when features are far from the expansion point, so the analysis is approximate in general cases.
- Off-diagonal Hessian effects are only partially blocked, and the theory relies on the standardization phase analysis with epsilon set to zero.
Experimental setup
Datasets: CIFAR-10 MNIST synthetic polynomial loss functions
Benchmarks: CIFAR-10 MNIST
Metrics: relative gradient difference (Delta grad) proportion of blocked loss terms (p_blocked) approximation error (p_error) gradient norm of L_linear t-SNE feature clustering RealNVP log-likelihood of interpolated images
Baselines: VGG-16 without BN DNN-LN RealNVP-LN ResNet-34 DenseNet-169
Builds on: batch normalization layer normalization RealNVP
Closest papers by meaning
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.90)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.88)
- Delving Into the Estimation Shift of Batch Normalization in a Network (CVPR 2022 · similarity 0.88)
- Batch normalization is sufficient for universal function approximation in CNNs (ICLR 2024 · similarity 0.88)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.87)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.