Research map / Architectures and efficiency / RNNs and normalization
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
Zhaodong Chen, Lei Deng, Bangyan Wang, Guoqi Li, Yuan Xie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use leaky ReLU gamma=0.18 instead of ReLU | Gaussian-init serial net · CIFAR-10 | test accuracy: +2.83% | Improved |
| raise leaky ReLU slope from 0.18 to 0.5 under BN (ablation) | leaky ReLU serial net · CIFAR-10 | test accuracy: +0.59% | Mixed |
| replace PReLU with sPReLU on delta orthogonal weights | serial net · CIFAR-10 | test accuracy: +7.73% | Improved |
| use SeLU eps=0.03 instead of original config | SeLU serial net · CIFAR-10 | test accuracy: +0.42% and +0.39% | Improved |
| compute second moment from pre-activations (SMN) instead of weights (sWS) (ablation) | normalization serial net · CIFAR-10 | test accuracy: 88.06% -> 88.50% | Improved |
| use SMN instead of BN | DenseNet · CIFAR-10 | test accuracy: 92.10% -> 92.06% | No clear effect |
| add mixup to SMN (ablation) | ResNet-56 · CIFAR-10 | test accuracy: 93.28% -> 93.93% | Improved |
| use Fixup p=1.5 instead of p=2 (ablation) | Fixup ResNet-56 with b&s and mixup · CIFAR-10 | test accuracy: 93.93% -> 94.28% | Improved |
| use SeLU eps=0.06 instead of original SeLU | Conv MobileNet V1 · ImageNet | top-1 error: Explode in the first epoch -> 30.37% | Improved |
| use SeLU eps=0.03 instead of 0.06 (ablation) | Conv MobileNet V1 · ImageNet | top-1 error: 30.37% -> 30.87% | Hurt |
| use leaky ReLU 0.3 instead of ReLU, Gaussian | Conv MobileNet V1 · ImageNet | top-1 error: 31.16% -> 29.39% | Improved |
| add mixup to SMN (ablation) | ResNet-50 · ImageNet | top-1 error: 24.90% -> 23.74% | Improved |
Closest papers by meaning
- Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice (NeurIPS 2017 · similarity 0.90)
- Norm matters: efficient and accurate normalization schemes in deep networks (NeurIPS 2018 · similarity 0.89)
- Scalable Optimization in the Modular Norm (NeurIPS 2024 · similarity 0.89)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.88)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.87)
- A path-norm toolkit for modern networks: consequences, promises and challenges (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
