Research map / Architectures and efficiency / RNNs and normalization
Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace small-batch BN with MABN | ResNet-50 · ImageNet val, normalization batch size 2 | top-1 error (%): 35.22 -> 23.58 | Helped |
| replace small-batch BN with MABN | Mask R-CNN with ResNet-50 FPN · COCO detection from scratch | AP bbox: 30.41 -> 34.85 | Helped |
| replace small-batch BN with MABN | Mask R-CNN with ResNet-50 FPN · COCO segmentation from scratch | AP mask: 27.91 -> 31.61 | Helped |
| replace BN with MABN | PSPNet with ResNet-101 · Cityscapes test | mIoU: 77.11 -> 78.20 | Helped |
| add EMAS to forward propagation (ablation) | modified normalization · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 29.68 -> 27.03 | Helped |
| add SMAS to backward propagation (ablation) | modified normalization with EMAS in FP · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 27.03 -> 23.58 | Helped |
| add EMAS to forward propagation (ablation) | vanilla normalization · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 35.22 -> 30.29 | Helped |
| add SMAS to backward propagation while retaining vanilla normalization (ablation) | BRN · ImageNet val, ResNet-50, normalization batch size 2 | training convergence | Hurt |
| use modified normalization instead of vanilla normalization (ablation) | ResNet-50 with regular batch size · ImageNet val, normalization batch size 32 | top-1 error (%): 23.41 -> 23.53 | No clear effect |
| use EMAS instead of SMAS in backward propagation (ablation) | modified normalization with EMAS in forward propagation · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 23.58 -> 24.83 | Hurt |
| use SMAS instead of EMAS in forward propagation (ablation) | modified normalization with SMAS in backward propagation · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 23.58 -> 26.63 | Hurt |
| remove weight centralization (ablation) | MABN · ImageNet val, ResNet-50, normalization batch size 2 | top-1 error (%): 23.58 -> 25.45 | Hurt |
Closest papers by meaning
- Double Forward Propagation for Memorized Batch Normalization (AAAI 2018 · similarity 0.93)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.92)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.91)
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.91)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.88)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.