Research map / Architectures and efficiency / RNNs and normalization
Training Faster by Separating Modes of Variation in Batch-Normalized Models
Mahdi M. Kalayeh, Mubarak Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BN in conv3 with mixture normalization | CIFAR CNN · CIFAR-10 (lr 0.01, wd 2e-5, 100 epochs) | test accuracy (%): 87.95 -> 88.47 | Improved |
| replace BN in conv3 with mixture normalization at large learning rate | CIFAR CNN · CIFAR-100 (lr 0.05, wd 1e-4) | test accuracy (%): 36.80 -> 43.77 | Improved |
| replace BN in conv3 with mixture normalization | CIFAR CNN · CIFAR-100 (lr 0.01, wd 1e-4) | test accuracy (%): 62.11 -> 62.20 | No clear effect |
| replace BN in conv3 with mixture normalization | CIFAR CNN · CIFAR-10 (lr 0.05, wd 1e-4) | test accuracy (%): 77.35 -> 77.33 | No clear effect |
| increase EM iterations from 2 to 8 (ablation) | MN in conv3 · CIFAR-100 CIFAR CNN | test accuracy (%): 62.29 -> 63.12 | Improved |
| increase EM iterations from 2 to 4 (ablation) | MN in conv3 · CIFAR-100 CIFAR CNN | test accuracy (%): 62.29 -> 62.76 | Improved |
| apply MN to conv2 and conv3 instead of conv3 only (ablation) | CIFAR CNN · CIFAR-100 | test accuracy (%): 62.29 -> 62.96 | Improved |
| use mixture normalization instead of BN (training steps to match BN accuracy) | CIFAR CNN · CIFAR-100 | steps to 62.14%: 1.81 x 10^4 -> 1.07 x 10^4 | Improved |
| use mixture normalization in Inception-V3 (steps to 91.50%) | Inception-V3 · CIFAR-10 | training steps: 7.34 x 10^4 -> 3.58 x 10^4 | Improved |
| apply MN to early reduction modules only (red1, red2) | Inception-V3 · CIFAR-100 | test accuracy (%): 71.30 -> 70.43 | Hurt |
| increase mixture components from K=3 to K=5 (ablation) | Inception-V3 MN · CIFAR-10 | test accuracy (%): 92.30 -> 92.25 | No clear effect |
| use mixture normalization at large learning rate 0.005 | Inception-V3 · CIFAR-100 | test accuracy (%): 52.63 -> 59.10 | Improved |
Closest papers by meaning
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.92)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.91)
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (ICLR 2017 · similarity 0.91)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.91)
- Re-Thinking the Effectiveness of Batch Normalization and Beyond (TPAMI 2023 · similarity 0.91)
- Kalman Normalization: Normalizing Internal Representations Across Network Layers (NeurIPS 2018 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
