Research map / Architectures and efficiency / RNNs and normalization
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize almost all weights to zero and use nondeterministic GPU operations for symmetry breaking | ConstNet · CIFAR-10 | accuracy: $95.40 \pm 0.07$ -> $95.37 \pm 0.06$ | No clear effect |
| Add 1% Dropout to zero-initialized training to break symmetry (ablation) | ConstNet · CIFAR-10 | accuracy: $95.37 \pm 0.06$ -> $95.46 \pm 0.13$ | No clear effect |
| Disable stochastic symmetry breaking by making computation deterministic (ablation) | Zero-initialized ConstNet · CIFAR-10 | accuracy: $95.37 \pm 0.06$ -> \red{24.79} $ \pm 0.58$ | Hurt |
| Replace I initialization with 𝟙 initialization in one layer (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $94.34\pm0.12$ | No clear effect |
| Replace I initialization with 𝟙 initialization in three layers (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $93.47\pm0.32$ | Hurt |
| Replace I initialization with 𝟙 initialization in five layers (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $87.85\pm0.52$ | Hurt |
| Replace I initialization with 𝟙 initialization in seven layers (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $44.89\pm2.52$ | Hurt |
| Use 𝟙 initialization instead of I initialization in every layer (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $29.77\pm2.47$ | Hurt |
| Increase the number of 𝟙-initialized layers, which average channels and impede symmetry-breaking signals (ablation) | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $29.77\pm2.47$ | Hurt |
| Use He initialization rather than I initialization | LeakyNet · CIFAR-10 | accuracy: $94.05\pm0.23$ -> $95.57\pm0.12$ | Helped |
Closest papers by meaning
- Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning (NeurIPS 2024 · similarity 0.87)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.87)
- Initial Guessing Bias: How Untrained Networks Favor Some Classes (ICML 2024 · similarity 0.86)
- Neural Redshift: Random Networks are not Random Functions (CVPR 2024 · similarity 0.86)
- From Activation to Initialization: Scaling Insights for Optimizing Neural Fields (CVPR 2024 · similarity 0.84)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.