Research map / Architectures and efficiency / RNNs and normalization
Norm-Preservation: Why Residual Networks Can Become Extremely Deep?
Alireza Zaeemzadeh, Nazanin Rahnavard, Mubarak Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use modified transition block (convolution moved before skip connection) (ablation) | ResNet164 · CIFAR-100 | error (%): 24.33 -> 23.06 | Improved |
| add norm-preserving projection regularization on transition blocks (ablation) | ResNet164 with modified transition block · CIFAR-100 | error (%): 23.06 -> 22.61 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | pre-activation ResNet164 · CIFAR-10 | error (%): 5.46 -> 4.75 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | pre-activation ResNet1001 · CIFAR-10 | error (%): 4.62 -> 3.72 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | pre-activation ResNet1001 · CIFAR-100 | error (%): 22.71 -> 19.99 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | ErasedReLU ResNet164 · CIFAR-10 | error (%): 4.65 -> 4.53 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | ErasedReLU ResNet1001 · CIFAR-100 | error (%): 20.63 -> 18.12 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | ResNet20 · CIFAR-10 generalization gap | mean gap (%): 5.5 -> 2.3 | Improved |
| replace transition blocks with norm-preserving ones (ProcResNet) | ResNet164 · CIFAR-10 generalization gap | max gap (%): 18.7 -> 8.7 | Improved |
| add identity skip connections (ResNet vs Plain) | Plain83 · CIFAR-10 generalization gap | mean gap (%): 7.5 -> 5.1 | Improved |
| add identity skip connections (ResNet vs Plain) | Plain164 · CIFAR-10 training stability | training convergence: unstable after 10 epochs -> stable | Improved |
| use norm-preserving projection every 2 iterations instead of SVD | ProcResNet164 · CIFAR-10 | training time overhead (%): 53% -> 7.6% | Improved |
Closest papers by meaning
- Are ResNets Provably Better than Linear Predictors? (NeurIPS 2018 · similarity 0.91)
- Towards Understanding the Importance of Shortcut Connections in Residual Networks (NeurIPS 2019 · similarity 0.90)
- Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.90)
- Learning Strict Identity Mappings in Deep Residual Networks (CVPR 2018 · similarity 0.89)
- Are deep ResNets provably better than linear predictors? (NeurIPS 2019 · similarity 0.89)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
