Research map / Architectures and efficiency / RNNs and normalization
Differentiable Learning-to-Normalize via Switchable Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn per-layer mixtures of IN, LN, and BN statistics | ResNet50 · ImageNet val, (8,32) | top-1 accuracy: 76.4 -> 76.9 vs BN; 75.9 -> 76.9 vs GN | Helped |
| learn per-layer mixtures of IN, LN, and BN statistics | ResNet50 · ImageNet val, (8,2) | top-1 accuracy: 65.3 -> 75.6 vs BN; 75.9 -> 75.6 vs GN | Mixed |
| learn per-layer mixtures of IN, LN, and BN statistics | ResNet50 · COCO 2017 val, Faster R-CNN | AP: 32.7 -> 33.0 vs GN | Helped |
| learn per-layer mixtures of IN, LN, and BN statistics | ResNet50 trained from scratch · COCO 2017 val, Faster R-CNN | AP: 28.3 -> 29.5 vs GN; 20.0 -> 29.5 vs BN | Helped |
| select one normalizer per layer using argmax (ablation) | SN · ImageNet, ResNet50 (8,32) | top-1 accuracy: 76.9% -> 77.0% | No clear effect |
| give each of two channel groups its own normalizer weights (ablation) | SN · ImageNet, ResNet50 | top-1 accuracy: 77.2% | Helped |
| estimate test-time BN statistics by batch average rather than moving average (ablation) | SN · ImageNet, ResNet50 (8,32) | top-1 accuracy: 76.89% -> 76.90% (50k samples); 76.89% -> 76.92% (all training samples) | Helped |
| use SN instead of IN | ResNet50 · ImageNet (8,32) | top-1 accuracy: 71.6% -> 76.9% | Helped |
| use SN instead of LN | ResNet50 · ImageNet (8,32) | top-1 accuracy: 74.7% -> 76.9% | Helped |
| use SN instead of BKN | ResNet50 · ImageNet (8,32) | top-1 accuracy: 76.8% -> 76.9% | No clear effect |
| replace BN moving average with batch average (ablation) | BN · ImageNet, ResNet50 | top-1 accuracy | Helped |
| increase minibatch size so BN statistics are less noisy (ablation) | SN · ImageNet | BN importance weight | Mixed |
Closest papers by meaning
- SSN: Learning Sparse Switchable Normalization via SparsestMax (CVPR 2019 · similarity 0.91)
- Exemplar Normalization for Learning Deep Representation (CVPR 2020 · similarity 0.89)
- Differentiable Dynamic Normalization for Learning Deep Representation (ICML 2019 · similarity 0.88)
- Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning (NeurIPS 2021 · similarity 0.88)
- Self-Normalizing Neural Networks (NeurIPS 2017 · similarity 0.87)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.