Research map / Architectures and efficiency / RNNs and normalization
Iterative Normalization: Beyond Standardization Towards Efficient Whitening
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BN modules with IterNorm | Wide residual networks · CIFAR-10, WRN-28-10 | test error (%): 3.89 $\pm$ 0.13 -> 3.56 $\pm$ 0.12 | Helped |
| Replace BN modules with IterNorm | Wide residual networks · CIFAR-10, WRN-40-10 | test error (%): 3.82 $\pm$ 0.11 -> 3.59 $\pm$ 0.07 | Helped |
| Replace the first BN module with IterNorm | Res-18 · ILSVRC-2012 val | Top-1 error (%): 29.76 -> 29.34 | Helped |
| Add IterNorm after average pooling (ablation) | Res-18 with IterNorm-L1 · ILSVRC-2012 val | Top-1 error (%): 29.34 -> 28.86 | Helped |
| Add IterNorm after average pooling (ablation) | Res-18 with IterNorm-L1 · ILSVRC-2012 val | Top-5 error (%): 10.22 -> 10.08 | Helped |
| Add IterNorm after average pooling | Res-50 with IterNorm-L1 · ILSVRC-2012 val | Top-1 error (%): 23.28 -> 22.91 | Helped |
| Add IterNorm after average pooling | Res-101 with IterNorm-L1 · ILSVRC-2012 val | Top-5 error (%): 5.99 -> 5.94 | Helped |
| Replace only the first BN module rather than all BN modules (ablation) | Res-18 with IterNorm · ILSVRC-2012 val | Top-1 error (%) | No clear effect |
| Use 7 iterations instead of the selected intermediate iteration count (ablation) | IterNorm · CIFAR-10, VGG | training efficiency | Hurt |
| Use 7 iterations (ablation) | IterNorm · CIFAR-10, VGG | test error | Hurt |
| Use 1 or 9 iterations rather than an intermediate iteration count (ablation) | IterNorm · MNIST, 4-layer MLP | training efficiency | Hurt |
| Use 9 iterations (ablation) | IterNorm · MNIST, 4-layer MLP | test error | Hurt |
Closest papers by meaning
- Stochastic Whitening Batch Normalization (CVPR 2021 · similarity 0.95)
- Decorrelated Batch Normalization (CVPR 2018 · similarity 0.93)
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.90)
- An Investigation Into the Stochasticity of Batch Whitening (CVPR 2020 · similarity 0.90)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.89)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.