Research map / Architectures and efficiency / RNNs and normalization
Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a non-asymptotic analysis of early neuron alignment for two-layer ReLU networks trained by gradient flow from small balanced initialization on well-separated binary data, bounding alignment time by O(log n / sqrt(mu)).
Problem
Understanding gradient-flow training dynamics of two-layer ReLU networks for binary classification under small initialization, specifically how long neuron directional alignment takes and how it depends on the initialization scale and data separation, with prior work assuming vanishing initializatio
Approach
Tracks the activation-pattern evolution of neurons under well-separated (positively/negatively correlated same/different-label) data with finite small initialization, showing neurons align with positive or negative data centers within O(log n / sqrt(mu)) time and giving an explicit bound on the initialization scale; then shows loss decays as O(1/t) and the first-layer weights become approximately
Key results
- Theorem gives directional convergence of all neurons within t1 = O(log n / sqrt(mu)) time under initialization scale epsilon = O((1/sqrt(h)) exp(-(n/sqrt(mu)) log n)).
- Loss converges at O(1/t) rate after alignment, and the stable rank of W satisfies limsup ||W||_F^2/||W||_2^2 <= 2.
- On centered MNIST with h=50, neurons align with x+ and x- after about 2000 iterations; stable rank drops below 2 after about 3000 iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train both first- and second-layer weights rather than only the first layer | Two-layer ReLU network with small initialization · MNIST two-digit binary classification | two-phase training | Helped |
| Decrease data separability from $\sin(0.1)$ to $\sin(0.001)$ | Gradient descent, small initialization · Two positive data points | time to reach $\mathcal{S}_+$: at iteration 7000 | Hurt |
| Use orthogonal data ($\mu=0$) instead of positively correlated data | Gradient descent, small initialization · Two positive data points | final neuron alignment | Hurt |
| Train only the first-layer weights and keep the second-layer weights fixed (ablation) | Two-layer ReLU network with small initialization · MNIST two-digit binary classification | two-phase training | Hurt |
Surprising findings
- With orthogonal data (mu=0), positive neurons can end up aligned with individual data points rather than the positive cone, so the S+ region is no longer the destination of all positive neurons.
- Training only the first layer does not exhibit the two-phase align-then-fit behavior.
Limitations
- Analysis requires well-separated data assumption (positive pairwise correlation within label, negative across labels) and balanced initialization, which MNIST only approximately satisfies after centering.
- Initialization scale bound is exponentially small in n, which is impractical.
- Results are for gradient flow and exponential/logistic loss with ReLU sign convention fixed; extension to differential inclusion only in appendix.
Experimental setup
Datasets: MNIST (two-digit binary subsets, centered) toy 2D dataset
Benchmarks: MNIST binary classification (digits 1 vs 7, 3 vs 8, and the main pair)
Metrics: training loss cosine alignment of neurons with x+ and x- stable rank
Baselines: Maennel et al. 2018 alignment analysis Phuong and Lampert 2021 orthogonal data analysis first-layer-only training (Boursier et al.)
Builds on: align then fit dynamics (Maennel et al. 2018 style analysis) orthogonally separable data analysis (Phuong and Lampert style) two-layer ReLU small initialization analysis (Luo et al. / Boursier et al.)
Closest papers by meaning
- Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs (NeurIPS 2023 · similarity 0.89)
- Convergence of Near-Linear Width ReLU Networks with Unbalanced Initialization (NeurIPS 2026 · similarity 0.89)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.89)
- Feature selection and low test error in shallow low-rotation ReLU networks (ICLR 2023 · similarity 0.88)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.87)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.