Research map / Architectures and efficiency / RNNs and normalization
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NormProp instead of Batch Normalization | Network in Network · CIFAR-10 without data augmentation | test error (%): 9.41 -> 9.11 | Helped |
| Use NormProp instead of Batch Normalization | Network in Network · CIFAR-10 with data augmentation | test error (%): 7.25 -> 7.47 | Hurt |
| Use NormProp instead of Batch Normalization | Network in Network · CIFAR-100 without data augmentation | test error (%): 35.32 -> 32.19 | Helped |
| Use NormProp instead of Batch Normalization | Network in Network · CIFAR-100 with data augmentation | test error (%): 30.26 -> 29.24 | Helped |
| Use NormProp instead of Batch Normalization | Network in Network · SVHN | test error (%): 2.25 -> 1.88 | Helped |
| Replace batch-dependent hidden-layer statistics with parametric normalization propagation | NormProp · CIFAR-10 validation convergence | hidden-layer input mean: avg. 0.33 -> avg. 0.19 | Helped |
| Replace batch-dependent hidden-layer statistics with parametric normalization propagation | NormProp · CIFAR-10 training convergence | convergence stability | Helped |
| Use NormProp rather than Batch Normalization | Network in Network · CIFAR-10, one training epoch | training time: ∼ 96 sec -> ∼ 84 sec | Helped |
| Switch between Global Data Normalization and Batch Data Normalization (ablation) | NormProp · CIFAR-10 | accuracy: 90.35% | No clear effect |
| Reduce training batch size, including to batch size 1 (ablation) | NormProp · CIFAR-10 | validation performance | Mixed |
| Add dropout (ablation) | NormProp · training | performance | No clear effect |
| Use SGD without momentum instead of SGD with momentum (ablation) | NormProp · SVHN and CIFAR-100 | performance | Hurt |
Closest papers by meaning
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.92)
- Double Forward Propagation for Memorized Batch Normalization (AAAI 2018 · similarity 0.91)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.91)
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization (ICLR 2020 · similarity 0.91)
- How Does Batch Normalization Help Optimization? (NeurIPS 2018 · similarity 0.90)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.