Research map / Architectures and efficiency / RNNs and normalization
Stochastic Whitening Batch Normalization
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BN layers with SWBN-KL | ResNetV2-164 · CIFAR-100 | accuracy: 75.56 -> 76.13 | Helped |
| Replace BN layers with SWBN-KL | ResNetV2-164 · CIFAR-10 | accuracy: 94.54 -> 94.68 | Helped |
| Replace BN layers with SWBN-Fro | ResNetV2-50 · ILSVRC-2012 | top-1 accuracy: 75.3 -> 76.95 | No clear effect |
| Replace BN layers with SWBN-KL | ResNeXt-101 · ILSVRC-2012 | top-1 accuracy: 78.8 -> 79.39 | Helped |
| Replace BN layers with SWBN-Fro | Matching Network / ResNet20 · CIFAR-FS 5-way 1-shot | accuracy: 61.28 -> 62.73 | Helped |
| Replace BN layers with SWBN-Fro | Prototypical Network / ResNet20 · CIFAR-FS 5-way 5-shot | accuracy: 73.47 -> 75.02 | Helped |
| Replace BN layers with IterNorm | ResNetV2-164 · CIFAR-100 | accuracy: 75.56 -> 74.12 | Hurt |
| Use SWBN-Fro instead of SWBN-KL (ablation) | VGG · CIFAR-10, whitening step size 10^-4 | training loss stability | Helped |
| Set whitening step size to 10^-6 instead of 10^-5 (ablation) | SWBN · CIFAR-10 | test loss | Hurt |
| Set whitening step size to 10^-4 instead of 10^-5 (ablation) | SWBN · CIFAR-10 | training loss stability | Hurt |
| Use SWBN-Fro instead of BN or IterNorm (ablation) | classification models · CIFAR-10, batch sizes 32, 128, and 512 | test loss | Helped |
| Replace BN layers with SWBN-KL or SWBN-Fro (ablation) | VGG · CIFAR-10 feature correlations | feature whitening | Helped |
Closest papers by meaning
- Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019 · similarity 0.95)
- Decorrelated Batch Normalization (CVPR 2018 · similarity 0.92)
- An Investigation Into the Stochasticity of Batch Whitening (CVPR 2020 · similarity 0.91)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.90)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.90)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.