Research map / Architectures and efficiency / RNNs and normalization
Characterizing signal propagation to close the performance gap in unnormalized ResNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Scaled Weight Standardization instead of Gaussian He-initialized weights | Normalizer-Free ResNets · ImageNet classification and initialization signal propagation | Top-1 accuracy and channel squared mean | Helped |
| Train NF-ResNet-50 with batch size 8 rather than batch size 1024 | NF-ResNet-50 · ImageNet, 15 epochs | Top-1 accuracy (%): 69.9 ± 0.1 -> 69.6 ± 0.1 | No clear effect |
| Train NF-ResNet-50 with batch size 4 rather than batch size 1024 | NF-ResNet-50 · ImageNet, 15 epochs | Top-1 accuracy (%): 69.9 ± 0.1 -> 69.9 ± 0.1 | No clear effect |
| Train BN-ResNet-50 with batch size 8 rather than batch size 1024 | BN-ResNet-50 · ImageNet, 15 epochs | Top-1 accuracy (%): 70.9 ± 0.1 -> 65.7 ± 0.2 | Hurt |
| Train BN-ResNet-50 with batch size 4 rather than batch size 1024 | BN-ResNet-50 · ImageNet, 15 epochs | Top-1 accuracy (%): 70.9 ± 0.1 -> 55.7 ± 0.3 | Hurt |
| Remove Scaled Weight Standardization from depth-wise convolutions (ablation) | Normalizer-Free EfficientNets · ImageNet | Top-1 accuracy | Helped |
| Apply a factor of 2 to Squeeze-and-Excitation output (ablation) | NF-RegNet · ImageNet training stability | Training stability | Helped |
| Replace strided 1x1 convolutions with average pooling followed by 1x1 convolutions (ablation) | NF-RegNet · ImageNet | Performance | Helped |
| Add a final expansion convolution after the last residual block (ablation) | NF-RegNet · ImageNet | Performance | Helped |
| Add a linear expansion layer after global average pooling (ablation) | NF-RegNet · ImageNet | Performance | No clear effect |
| Use inverted bottlenecks rather than compressive bottlenecks (ablation) | NF-RegNet · ImageNet | Performance | Helped |
| Use SiLU with EMA rather than ReLU (ablation) | NF-RegNet · ImageNet | Performance | Helped |
Closest papers by meaning
- Exploring Normalization in Deep Residual Networks with Concatenated Rectified Linear Units (AAAI 2017 · similarity 0.90)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.89)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.89)
- High-Performance Large-Scale Image Recognition Without Normalization (ICML 2021 · similarity 0.88)
- A Mean Field Theory of Batch Normalization (ICLR 2019 · similarity 0.87)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.