Research map / Architectures and efficiency / RNNs and normalization
Deep ReLU Networks Preserve Expected Length
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard He initialization with weight variance 2/fan-in | Fully connected ReLU networks · Unit-length curves at initialization | expected length distortion: \E{\text{length distortion}}\leq \sqrt{n_L/n_0} | No clear effect |
| Increase depth while keeping hidden widths fixed | Fully connected ReLU networks with He initialization · Unit-length curves at initialization | expected length distortion: -\frac{5}{8}\sum_{\ell=1}^{L-1}n_\ell^{-1} | Hurt |
| Use narrower hidden layers | Fully connected ReLU networks with He initialization · Unit-length curves at initialization | expected length distortion: -\frac{5}{8}\sum_{\ell=1}^{L-1}n_\ell^{-1} | Hurt |
| Scale weights and biases by c | Depth-L ReLU networks · Length distortion at initialization | length distortion: c^L | Mixed |
| Use He initialization in residual networks with residual-module scales 1/L | Residual networks · Unit-length line segment at initialization | mean length distortion | Hurt |
| Use He initialization in sequential convolutional networks | Convolutional networks · Unit-length line segment at initialization | mean length distortion: \sqrt{n_L/n_0}\approx 0.08 | Hurt |
Closest papers by meaning
- The Expressive Power of Neural Networks: A View from the Width (NeurIPS 2017 · similarity 0.90)
- Finite Depth and Width Corrections to the Neural Tangent Kernel (ICLR 2020 · similarity 0.90)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.88)
- On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.88)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.87)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.