Research map / Architectures and efficiency / RNNs and normalization
Momentum Batch Normalization for Deep Learning with Small Batch Size
Hongwei Yong, Jianqiang Huang, Deyu Meng, Xiansheng Hua, Lei Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Batch Renormalization with momentum BN | BN with small batch · Mini-ImageNet-100 (batch size 2 per GPU) | accuracy (%): 55.47% -> 56.50% | Improved |
| replace BN with MBN | ResNet50 · ImageNet val (64 GPUs, batch 4 per GPU) | accuracy: +2.5% | Improved |
| use inference momentum tau=0.99 (adaptive to batch size) instead of 0.9 (ablation) | BN and MBN, batch 2 per GPU · CIFAR100 | test accuracy | Improved |
| set m0 very small (4) in MBN (ablation) | MBN ResNet18 · CIFAR100 | test accuracy | Hurt |
| vary m0 from 16 to 128 (ablation) | MBN ResNet18 · CIFAR100 | test accuracy | No clear effect |
| replace BN with MBN at batch size 2 per GPU | ResNet18 · CIFAR10/CIFAR100 | test accuracy | Improved |
| replace BN with MBN at batch size 8 per GPU | ResNet18 · CIFAR10/CIFAR100 | test accuracy | No clear effect |
| replace BN with MBN on ResNet34, ResNet50, VGG11, VGG16 | several networks, batch 2 per GPU · CIFAR100 | test accuracy | Improved |
Closest papers by meaning
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.93)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.92)
- Re-Thinking the Effectiveness of Batch Normalization and Beyond (TPAMI 2023 · similarity 0.91)
- Double Forward Propagation for Memorized Batch Normalization (AAAI 2018 · similarity 0.91)
- Revisiting Batch Norm Initialization (ECCV 2022 · similarity 0.91)
- Ghost Noise for Regularizing Deep Neural Networks (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
