Research map / Architectures and efficiency / RNNs and normalization
Critical Initialization of Wide and Deep Neural Networks using Partial Jacobians: General Theory and Applications
Darshil Doshi, Tianyu He, Andrey Gromov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add residual connections with strength mu=1 to LayerNorm MLP (ablation) | deep MLP (L=50, N=500) · FashionMNIST | training accuracy | Improved |
| use erf activation with mu=1 residual connections without LayerNorm (subcritical case) (ablation) | deep MLP (L=50, N=500) · FashionMNIST | training accuracy | Improved |
| decrease sigma_w in erf MLP with mu=1 residual connections (ablation) | deep MLP with erf · FashionMNIST | training accuracy | Improved |
| increase residual strength mu (ablation) | ResNet110 (LayerNorm) · training accuracy vs sigma_w^2 | training accuracy | Improved |
| increase mu monotonically at (sigma_w^2, sigma_b^2) = (2,0) (ablation) | ResNet110 (LayerNorm) · training accuracy vs mu | training accuracy | Improved |
| set mu=0 with small sigma_w^2 (ablation) | ResNet110 (LayerNorm) · training accuracy | training accuracy | Hurt |
| increase mu from 0.5 to 1 (ablation) | MLP-Mixer (L=100 blocks) · CIFAR-10 training curves | training accuracy: 0.5 -> 1 | Improved |
| use mu=1 residual connection with BatchNorm | deep network · trainability | trainability | Improved |
Closest papers by meaning
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.89)
- Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limit (JMLR 2024 · similarity 0.88)
- Expected Gradients of Maxout Networks and Consequences to Parameter Initialization (ICML 2023 · similarity 0.88)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.87)
- Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes (ICLR 2024 · similarity 0.87)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
