Research map / Architectures and efficiency / RNNs and normalization
Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reparameterizes a 4-neuron-wide ReLU network with triangle-wave peak parameters and scaling coefficients so that depth-d networks start with 2^d linear regions, then pretrains and trains to approximate 1D convex functions orders of magnitude more accurately than Kaiming or RAAI initializations.
Problem
Randomly initialized ReLU networks rarely realize their exponential number of linear regions, leading to oversized networks and poor function approximation, with gradient descent unable to directly create new linear regions.
Approach
A trainable reparameterization that ties a 4-neuron-wide deep ReLU network's weights to triangle-wave peak locations and a differentiability-enforcing scaling rule, so any peak values yield 2^d linear regions; a pretraining phase optimizes the peaks and then switches to standard weight parameterization.
Key results
- On x^3 and x^11 approximation, pretraining with Theorem 3.1 reaches minimum MSE 7.86e-8 (x^3) vs 2.11e-5 for Kaiming default, roughly three orders of magnitude lower.
- On sin(x) and tanh(3x), the best pretrained variant reaches minimum MSE 5.06e-8 and 6.82e-8 vs 4.50e-5 and 5.75e-5 for Kaiming default.
- On ImageNet VGG-16 classifier retraining, the modified network has an early-training advantage but fully trained accuracies are comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use triangle-based initialization and pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · x^3, 500-point training set | minimum MSE: 2.11 \times 10^{-5} -> 7.86 \times 10^{-8} | Helped |
| use triangle-based initialization and pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · x^11, 500-point training set | minimum MSE: 2.19 \times 10^{-5} -> 8.86 \times 10^{-7} | Helped |
| use triangle-based initialization and pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · sin(x), 500-point training set | mean MSE: 1.15 \times 10^{-1} -> 2.21 \times 10^{-7} | Helped |
| use triangle-based initialization and pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · tanh(3x), 500-point training set | mean MSE: 1.96 \times 10^{-1} -> 8.42 \times 10^{-7} | Helped |
| skip pretraining and train only in the standard parameterization (ablation) | triangle-initialized ReLU network · x^3, 500-point training set | minimum MSE: 7.63 \times 10^{-7} -> 7.86 \times 10^{-8} | Hurt |
| skip pretraining and train only in the standard parameterization (ablation) | triangle-initialized ReLU network · x^3, 500-point training set | mean MSE: 3.89 \times 10^{-5} -> 5.27 \times 10^{-7} | Hurt |
| remove differentiability enforcement with Thm. 3.1 (ablation) | pretrained ReLU network · sin(x), 500-point training set | mean MSE: 1.47 \times 10^{-5} -> 2.21 \times 10^{-7} | Hurt |
| remove differentiability enforcement with Thm. 3.1 (ablation) | pretrained ReLU network · tanh(3x), 500-point training set | mean MSE: 3.81 \times 10^{-4} -> 8.42 \times 10^{-7} | Hurt |
| use triangle-based pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · x^3, sparse training data; unseen points | minimum MSE: 2.41 \times 10^{-4} -> 5.65 \times 10^{-6} | Helped |
| use triangle-based pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · x^11, sparse training data; unseen points | minimum MSE: 2.14 \times 10^{-3} -> 6.53 \times 10^{-4} | Helped |
| use triangle-based pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · sin(x), sparse training data; unseen points | minimum MSE: 2.27 \times 10^{-5} -> 7.92 \times 10^{-7} | Helped |
| use triangle-based pretraining with differentiability enforced by Thm. 3.1 | 4-neuron-wide ReLU network · tanh(3x), sparse training data; unseen points | minimum MSE: 1.60 \times 10^{-4} -> 5.09 \times 10^{-6} | Helped |
Surprising findings
- Roughly half of default Kaiming-initialized networks collapse from the dying ReLU phenomenon in these 1D experiments.
- Switching parameterization made no difference for the VGG-16 classification task, though it was critical for 1D regression.
Limitations
- Construction limited to 4-neuron-wide networks and to 1D convex functions; multidimensional and non-convex cases rely on combining multiple subnetworks.
- Classification gains on ImageNet are not significant; the method may need practical-scale regression tasks to show advantage.
- Sparse block-diagonal structure leaves most dense-layer weights unused in the VGG experiment.
Experimental setup
Datasets: ImageNet synthetic 1D function samples (x^3, x^11, tanh(3x), sin(x), x^3-x, r^3)
Benchmarks: ImageNet 1D function approximation test sets
Metrics: mean squared error minimum MSE over 30 trials top-1 accuracy
Baselines: Kaiming default initialization RAAI distribution pretraining skipped pretraining unregularized VGG-16 standard classifier
Builds on: Telgarsky triangle-wave construction Yarotsky-style x^2 approximation Kaiming initialization RAAI initialization Adam
Closest papers by meaning
- Understanding Deep Neural Networks with Rectified Linear Units (ICLR 2018 · similarity 0.91)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.90)
- On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths (ICML 2021 · similarity 0.90)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.89)
- On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.88)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.