Research map / Architectures and efficiency / RNNs and normalization
Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives precise finite-width formulas (via mixed ReLU-Gaussian moments J_{a,b} and Bessel numbers) for how the angle between two inputs collapses with depth in fully connected ReLU networks at He initialization, beyond the infinite-width prediction.
Problem
Depth degeneracy in fully connected ReLU networks at initialization: inputs become nearly collinear as depth grows, but infinite-width theory predicts only a polynomial (ell^-1) rate and zero variance, missing finite-width fluctuations.
Approach
Treats the log-squared sine of the angle as a Markov chain and uses combinatorial expansions of sums over neuron pairs to get finite-width mean and variance in closed form; introduces a Gaussian integration-by-parts recurrence for mixed ReLU moments J_{a,b} that is solved explicitly via Bessel numbers.
Key results
- Approximation ln sin^2(theta_{l+1}) ≈ ln sin^2(theta_l) - (2/(3π))theta_l - rho(n_l) with rho(n)=2/n+O(n^-2) predicts exponential decay of theta in depth driven by finite width, unlike infinite-width ell^-1 decay.
- Variance of ln sin^2(theta_{l+1}) is approximately 8/n at small angles, a quantity infinite-width theory predicts as zero.
- Explicit Bessel-number formula gives closed-form J_{a,b}(theta) for mixed ReLU-Gaussian moments, with validation against Monte Carlo simulations of networks with width 1024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase network depth under standard ReLU initialization | Fully connected ReLU network · Initialization | input angle: θ_ℓ ≤ exp(-1/2 ∑_{i=1}^ℓ ρ(n_i)) | Hurt |
| increase layer width | Fully connected ReLU network · Initialization | input angle: ρ(n) = 2/n + O(n^{-2}) | Helped |
| use the finite-width angle prediction instead of the infinite-width update rule | Angle prediction algorithm · Finite-network Monte Carlo simulations | prediction accuracy | Helped |
| select architectures whose predicted initialization angle is small | Neural architecture search · After-training test performance | test accuracy | Hurt |
Surprising findings
- Finite-width depth degeneracy is exponentially fast in depth (through sum of 1/n_i), in contrast to the polynomial rate predicted by infinite-width theory.
- The mixed moments J_{a,b} connect combinatorially to Bessel numbers, discovered via OEIS lookup.
Limitations
- Approximations are valid only for small angles and large layer widths (theta << 1, n >> 1).
- Analysis restricted to fully connected ReLU networks at He initialization; extension to other nonlinearities and architectures is left open.
Experimental setup
Metrics: mean and variance of ln sin^2(theta) angle prediction error vs Monte Carlo
Baselines: infinite-width angle prediction (Cho-Saul / edge of chaos)
Builds on: Cho and Saul arc-cosine kernel (J_p functions) He initialization Edge of chaos (Poole et al. / Schoenholz et al.)
Closest papers by meaning
- Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depth (NeurIPS 2020 · similarity 0.87)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.86)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.85)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.85)
- Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.84)
- The Symmetries of Three-Layer ReLU Networks (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.