AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Norm-Preservation: Why Residual Networks Can Become Extremely Deep?

Alireza Zaeemzadeh, Nazanin Rahnavard, Mubarak Shah

TPAMI 2020 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 109 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use modified transition block (convolution moved before skip connection) (ablation)ResNet164 · CIFAR-100error (%): 24.33 -> 23.06Improved
add norm-preserving projection regularization on transition blocks (ablation)ResNet164 with modified transition block · CIFAR-100error (%): 23.06 -> 22.61Improved
replace transition blocks with norm-preserving ones (ProcResNet)pre-activation ResNet164 · CIFAR-10error (%): 5.46 -> 4.75Improved
replace transition blocks with norm-preserving ones (ProcResNet)pre-activation ResNet1001 · CIFAR-10error (%): 4.62 -> 3.72Improved
replace transition blocks with norm-preserving ones (ProcResNet)pre-activation ResNet1001 · CIFAR-100error (%): 22.71 -> 19.99Improved
replace transition blocks with norm-preserving ones (ProcResNet)ErasedReLU ResNet164 · CIFAR-10error (%): 4.65 -> 4.53Improved
replace transition blocks with norm-preserving ones (ProcResNet)ErasedReLU ResNet1001 · CIFAR-100error (%): 20.63 -> 18.12Improved
replace transition blocks with norm-preserving ones (ProcResNet)ResNet20 · CIFAR-10 generalization gapmean gap (%): 5.5 -> 2.3Improved
replace transition blocks with norm-preserving ones (ProcResNet)ResNet164 · CIFAR-10 generalization gapmax gap (%): 18.7 -> 8.7Improved
add identity skip connections (ResNet vs Plain)Plain83 · CIFAR-10 generalization gapmean gap (%): 7.5 -> 5.1Improved
add identity skip connections (ResNet vs Plain)Plain164 · CIFAR-10 training stabilitytraining convergence: unstable after 10 epochs -> stableImproved
use norm-preserving projection every 2 iterations instead of SVDProcResNet164 · CIFAR-10training time overhead (%): 53% -> 7.6%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy