Research map / Architectures and efficiency / RNNs and normalization
Switchable Normalization for Learning-to-Normalize Deep Representation
Ping Luo, Ruimao Zhang, Jiamin Ren, Zhanglin Peng, Jingyu Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BN with Switchable Normalization | ResNet50 · ImageNet val, batch (8,32) | top-1 accuracy (%): 76.4 -> 76.9 | Improved |
| replace BN with SN at tiny minibatch | ResNet50 · ImageNet val, batch (8,2) | top-1 accuracy (%): 65.3 -> 75.6 | Improved |
| replace GN with SN at small minibatch | ResNet50 · ImageNet val, batch (8,2) | top-1 accuracy (%): 75.9 -> 75.6 | No clear effect |
| replace BN with SN | ResNet101 · ImageNet val, batch (8,32) | top-1 accuracy (%): 77.6 -> 78.6 | Improved |
| use batch average instead of moving average for BN statistics in SN (ablation) | SN ResNet50 · ImageNet val | top-1 accuracy (%): 76.9 vs 76.89 | No clear effect |
| learn mixed normalizers via SN instead of IN (ablation) | ResNet50 · ImageNet val, batch (8,32) | top-1 accuracy (%): 71.6 -> 76.9 | Improved |
| replace LN with SN (ablation) | ResNet50 · ImageNet val, batch (8,32) | top-1 accuracy (%): 74.7 -> 76.9 | Improved |
| replace BRN with SN (ablation) | ResNet50 · ImageNet val, batch (8,4) | top-1 accuracy (%): 73.7 -> 75.9 | Improved |
| make SN sparse with argmax on control parameters (ablation) | SN ResNet50 · ImageNet val, batch (8,32) | top-1 accuracy (%): 76.9 -> 77.0 | No clear effect |
| finetune BN backbone in detection instead of freezing (ablation) | Faster R-CNN ResNet50 · COCO | AP: 29.6 -> 19.3 | Hurt |
| replace GN with SN in backbone and head | Faster R-CNN+FPN · COCO | AP: 38.2 -> 39.3 | Improved |
| replace GN with SN in head over frozen BN backbone (ablation) | Faster R-CNN+FPN · COCO | AP: 37.2 -> 38.0 | Improved |
Closest papers by meaning
- Differentiable Learning-to-Normalize via Switchable Normalization (ICLR 2019 · similarity 0.98)
- SSN: Learning Sparse Switchable Normalization via SparsestMax (CVPR 2019 · similarity 0.91)
- Exemplar Normalization for Learning Deep Representation (CVPR 2020 · similarity 0.90)
- Self-Normalizing Neural Networks (NeurIPS 2017 · similarity 0.88)
- Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning (NeurIPS 2021 · similarity 0.88)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
