AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Devansh Arpit, Víctor Campos, Yoshua Bengio

NeurIPS 2019 · Architectures and efficiency › RNNs and normalization · approach: Normalization layer theory · 16 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use the proposed initialization instead of data-dependent initializationWeight-normalized ResNet-56 · CIFAR-10test error (%): 9.19 $\pm$ 0.24 -> 7.87 $\pm$ 0.14Improved
Add learning-rate warmup to proposed initialization (ablation)Weight-normalized ResNet-56 · CIFAR-10test error (%): 7.87 $\pm$ 0.14 -> 7.20 $\pm$ 0.12Improved
Use the proposed initialization instead of data-dependent initializationWeight-normalized ResNet-110 · CIFAR-10test error (%): 9.33 $\pm$ 0.10 -> 7.71 $\pm$ 0.14Improved
Add learning-rate warmup to proposed initialization (ablation)Weight-normalized ResNet-110 · CIFAR-10test error (%): 7.71 $\pm$ 0.14 -> 6.69 $\pm$ 0.11Improved
Use the proposed initialization instead of data-dependent initializationWeight-normalized WRN-40-10 with cutout · CIFAR-10test error (%): 6.10 $\pm$ 0.23 -> 4.74 $\pm$ 0.14Improved
Add learning-rate warmup to proposed initialization with cutout (ablation)Weight-normalized WRN-40-10 · CIFAR-10test error (%): 4.74 $\pm$ 0.14 -> 4.75 $\pm$ 0.08No clear effect
Use the proposed initialization instead of data-dependent initializationWeight-normalized ResNet-164 with cutout · CIFAR-100test error (%): 30.26 $\pm$ 0.51 -> 27.30 $\pm$ 0.49Improved
Add learning-rate warmup to proposed initialization with cutout (ablation)Weight-normalized ResNet-164 · CIFAR-100test error (%): 27.30 $\pm$ 0.49 -> 25.31 $\pm$ 0.26Improved
Use the proposed initialization instead of data-dependent initializationWeight-normalized residual networks · CIFAR-10 and CIFAR-100; test errortest error (%): 9.19 $\pm$ 0.24 -> 7.87 $\pm$ 0.14; 9.33 $\pm$ 0.10 -> 7.71 $\pm$ 0.14; 6.10 $\pm$ 0.23 -> 4.74 $\pm$ 0.14; 30.26 $\pm$ Improved
Use proposed initialization with warmup instead of data-dependent initializationWeight-normalized residual networks · CIFAR-10 ResNet-56 and ResNet-110; CIFAR-100 ResNet-164test error (%): 9.19 $\pm$ 0.24 -> 7.20 $\pm$ 0.12; 9.33 $\pm$ 0.10 -> 6.69 $\pm$ 0.11; 30.26 $\pm$ 0.51 -> 25.31 $\pm$ 0.26Improved
Use the proposed initialization rather than PyTorch defaultWeight-normalized residual networks · CIFAR-10 WRNs across evaluated depthssuccessful training across depthsImproved
Use the proposed initialization rather than data-dependent initializationWeight-normalized fully connected networks · MNIST, depths up to 200 layerstrainability at depthImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy