Research map / Architectures and efficiency / RNNs and normalization
Characterizing Well-Behaved vs. Pathological Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use batch-normalized residual connections instead of a batch-normalized feedforward network | Networks at initialization · CIFAR-10 | normalized sensitivity growth with depth | Helped |
| Use batch-normalized residual connections instead of a batch-normalized feedforward network | Networks at initialization · CIFAR-10 | signal conditioning | Helped |
| Use a vanilla feedforward network with standard initialization | Networks at initialization · CIFAR-10 | effective rank of signal | Hurt |
| Use a batch-normalized feedforward network | Networks at initialization · CIFAR-10 | normalized sensitivity | Hurt |
| Add skip connections with residual-path dilution (ablation) | Batch-normalized feedforward networks · CIFAR-10 | normalized sensitivity growth with depth | Helped |
| Remove residual-path dilution by multiplying the skip branch by 0 (ablation) | Batch-normalized ResNets · CIFAR-10 | normalized sensitivity growth with depth | Hurt |
| Use zero-padding instead of periodic convolution boundaries (ablation) | Vanilla networks · CIFAR-10 | effective rank of signal | Helped |
| Use reflective instead of periodic convolution boundaries (ablation) | Vanilla networks · CIFAR-10 | effective rank of signal | No clear effect |
| Use zero-padding or reflective instead of periodic convolution boundaries (ablation) | Batch-normalized feedforward networks and batch-normalized ResNets · CIFAR-10 | depth evolution | No clear effect |
| Change the dataset from CIFAR-10 to MNIST (ablation) | Vanilla networks, batch-normalized feedforward networks, and batch-normalized ResNets · MNIST | depth evolution | No clear effect |
| Set batch-normalization epsilon to 0 instead of 0.001 (ablation) | Batch-normalized feedforward networks · CIFAR-10 | depth evolution | No clear effect |
| Increase the number of input images used to estimate batch-normalization statistics (ablation) | Batch-normalized feedforward networks · CIFAR-10 | signal conditioning | Hurt |
Closest papers by meaning
- The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks (NeurIPS 2019 · similarity 0.88)
- A Mean Field Theory of Batch Normalization (ICLR 2019 · similarity 0.88)
- Batch normalization provably avoids ranks collapse for randomly initialised deep networks (NeurIPS 2020 · similarity 0.87)
- On the Expressive Power of Deep Neural Networks (ICML 2017 · similarity 0.87)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.86)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.