Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Critical initialisation for deep signal propagation in noisy rectifier neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use critical initialization that accounts for multiplicative-noise second moment | Noisy ReLU networks · MNIST and CIFAR-10, variance propagation | signal variance stability | Helped |
| Use He initialization without accounting for dropout noise | ReLU networks with dropout · Deep-network variance propagation | signal variance stability | Hurt |
| Use zero-mean additive noise | ReLU networks · Deep signal propagation | signal variance stability | Hurt |
| Increase multiplicative-noise strength | Critically initialized ReLU networks · Input correlation propagation | discriminatory information depth | Hurt |
| Use stronger dropout | Critically initialized ReLU networks · MNIST and CIFAR-10 | validation loss versus depth | Hurt |
| Increase layer widths with noise according to D_{l+1} = D_l\mu_2 | Critically initialized noisy ReLU networks · Backward gradient propagation | gradient variance stability | Helped |
Closest papers by meaning
- Critical initialisation in continuous approximations of binary neural networks (ICLR 2020 · similarity 0.87)
- Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers (ICLR 2019 · similarity 0.87)
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (ICML 2018 · similarity 0.87)
- Initialization of ReLUs for Dynamical Isometry (NeurIPS 2019 · similarity 0.87)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.85)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.