Research map / Optimization / Optimizers
Delving into Variance Transmission and Normalization: Shift of Average Gradient Makes the Network Collapse
Yuxiang Liu, Jidong Ge, Chuanyi Li, Jie Gui
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace output normalization with PWS on convolution filters | SSD · VOC07 test, batch size 2 | mAP: 72.97 -> 76.93 | Improved |
| Replace output normalization with PWS on convolution filters | SSD · VOC07 test, batch size 8 | mAP: 76.74 -> 77.66 | Improved |
| Use PWS instead of BN | Mask R-CNN with ResNet-50 · COCO, 1x schedule | AP bbox: 38.0 -> 38.9 | Improved |
| Use PWS instead of BN | SSD with VGG-16 · COCO, 2x schedule | AP bbox: 26.6 -> 27.2 | Improved |
| Replace BN with PWS | ResNet-50 · CIFAR-10 residual, LR 5e-2 | top-1 error: 9.14 -> 6.75 | Improved |
| Replace BN with PWS | ResNet-50 · ImageNet val | top-5 error: 23.6 -> 24.0 | Hurt |
| Add trainable alpha to PWS (ablation) | PWS with VGG-16 · VOC07 test, LR 1e-2 | mAP: 74.96 -> 77.52 | Improved |
| Add the sqrt(2/n_l) variance scaling factor (ablation) | PWS with VGG-16 · VOC07 test, LR 1e-2 | mAP: NaN -> 77.52 | Improved |
| Set gamma to 1e-3 instead of 1e-5 (ablation) | PWS with VGG-16 · VOC07 test, LR 1e-3 | mAP: 72.99 -> 70.32 | Hurt |
| Set gamma to 1e-5 instead of 1e-3 (ablation) | PWS with VGG-16 · VOC07 test, LR 1e-2 | mAP: 77.52 -> NaN | Hurt |
| Set gamma to 1e-4 instead of 1e-3 (ablation) | PWS with VGG-16 · VOC07 test, LR 1e-2 | mAP: 77.52 -> 76.50 | Hurt |
Closest papers by meaning
- Re-Thinking the Effectiveness of Batch Normalization and Beyond (TPAMI 2023 · similarity 0.88)
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (ICML 2016 · similarity 0.88)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.88)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.87)
- Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift (CVPR 2019 · similarity 0.87)
- Variance-Aware Weight Initialization for Point Convolutional Neural Networks (ECCV 2022 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
