Research map / Detection and segmentation / CNN features
Is normalization indispensable for training deep neural network?
Jie Shao, Kai Hu, Changhu Wang, Xiangyang Xue, Bhiksha Raj
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace no multiplier with a learnable scalar multiplier per residual block (ablation) | RescaleNet · ImageNet val | top-1 accuracy (%): 74.2 -> 76.6 | Improved |
| use scalar rather than vector multipliers (ablation) | RescaleNet · ImageNet val | top-1 accuracy (%): 74.8 -> 76.6 | Improved |
| set hyper-parameter c = L instead of c = 1 or L2 (ablation) | RescaleNet · ImageNet val | top-1 accuracy (%): 76.4 -> 76.6 | No clear effect |
| data-dependent bias initialization instead of zero init (ablation) | RescaleNet · ImageNet val | top-1 accuracy (%): 76.4 -> 76.6 | Improved |
| apply bias before weight multiplication instead of after (ablation) | RescaleNet · ImageNet val | top-1 accuracy (%): 76.1 -> 76.6 | Improved |
| add dropout to non-normalized RescaleNet | ResNet50 · ImageNet val | top-1 accuracy (%): 74.3 -> 76.6 | Improved |
| RescaleNet instead of Group Normalization | ResNet50 · ImageNet val | top-1 accuracy (%): 75.9 -> 76.58 | Improved |
| RescaleNet instead of Batch Normalization with 200-epoch refined recipe | ResNet-D · ImageNet val | top-1 accuracy (%): 79.15 -> 78.93 | No clear effect |
| RescaleNet instead of BN on SE-ResNet50 | SE-ResNet50 · ImageNet val | top-1 accuracy (%): 76.7 -> 77.22 | Improved |
| RescaleNet instead of Group Norm on Mask R-CNN FPN 3X | ResNet50 · COCO val2017 | AP bbox: 40.8 -> 41.0 | Improved |
| RescaleNet instead of Layer Norm in transformer | Transformer · IWSLT14 DE-EN | BLEU: 34.2 -> 35.0 | Improved |
| pre-weight bias only, without normalization, on VGG-19 | VGG-19 · ImageNet val | top-1 accuracy (%): 74.4 -> 75.0 | Improved |
Closest papers by meaning
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (TPAMI 2023 · similarity 0.89)
- New Interpretations of Normalization Methods in Deep Learning (AAAI 2020 · similarity 0.87)
- Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020 · similarity 0.87)
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (ICLR 2017 · similarity 0.86)
- Optimization Theory for ReLU Neural Networks Trained with Normalization Layers (ICML 2020 · similarity 0.86)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
