Research map / Optimization / Implicit bias and generalization
Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that flat/stable minima of two-layer ReLU networks generalize but their rates degrade exponentially with input dimension (curse of dimensionality), via a weighted variation norm, minimax lower bounds, and "neural shattering" experiments.
Problem
Whether stable (flat) minima of two-layer overparameterized ReLU networks generalize and estimate functions well in the high-dimensional, non-interpolation regime, including on the full input domain.
Approach
Shows every stable solution lies in a data-dependent weighted variation space (weight g(u,t) from loss Hessian), derives generalization and MSE upper bounds on the full domain, and builds a sphere-cap packing of boundary-localized ReLU neurons giving matching-style minimax lower bounds that exhibit exponential dimension dependence; experiments reveal "neural shattering" where neurons activate rare
Key results
- Theory: stable minima achieve MSE rate n^{-1/(2d+4)} up to logs, while the minimax lower bound is n^{-2/(d+1)} for d>1 (n^{-1/2} for d=1), showing exponential dimension dependence versus n^{-1/4} for low-norm solutions.
- Experiments: slope of log MSE vs log n for GD with large learning rate falls to about 0.1 at d=5, while weight-decay solutions keep slopes above 0.5.
- Neural shattering: with eta=0.9 on 10-d linear target, neuron activation rates drop to at most 10% while weights stay large, giving train MSE about 1.105 vs 0.055 with weight decay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with a large learning rate and no weight decay instead of ℓ² weight decay | Two-layer ReLU network · 10D linear target, 512 noisy samples | train MSE: ≈ 0.055 -> ≈ 1.105 | Hurt |
| Increase input dimension | Vanilla gradient descent · Uniform unit ball, dimensions 1–5 | MSE versus sample-size slope: falling to about 0.1 at d=5 | Hurt |
| Use ℓ² weight decay instead of vanilla gradient descent | Two-layer ReLU network · Uniform unit ball, dimensions 1–5 | MSE versus sample-size slope: above 0.5 | Helped |
| Use stable minima in higher-dimensional nonparametric estimation | Two-layer ReLU network · Uniform unit ball, d>1 | MSE rate: \left(\frac{\sigma^2}{n}\right)^{\frac{1}{2d+4}} | Hurt |
| Use stable minima rather than low-norm solutions | Two-layer ReLU network · Uniform unit ball | generalization gap rate: n^{-\frac{1}{2d+4}}; \widetilde{O}(n^{-\frac{1}{4}}) | Hurt |
Surprising findings
- Flat minima found by large-step GD can overfit badly in high dimensions, with neurons activating on very few points yet keeping large weights (neural shattering).
- Fraction of surviving neurons with near-zero activation rises with input dimension (up to 65% at d=6 in the appendix).
Limitations
- Theory restricted to two-layer ReLU networks and uniform distribution on the unit ball.
- Upper bounds hide logarithmic and dimension-dependent constants; the gap between upper and lower bounds remains.
- Experiments are small-scale synthetic regression with linear and Hölder targets.
Experimental setup
Datasets: synthetic uniform-ball regression data with linear and Hölder(1/2) targets
Benchmarks: synthetic regression (MSE vs sample size, dimensions 1-5)
Metrics: mean squared error log-log slope of MSE vs n per-neuron activation rate vs weight norm
Baselines: weight decay / low-norm solutions
Builds on: Implicit bias of minima stability (Mulayoff & Michaeli style analysis) Variation spaces / path norm for ReLU networks Edge of stability (Cohen et al.) Radon-domain total variation (Ongie et al.)
Closest papers by meaning
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.93)
- The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks (ICLR 2023 · similarity 0.93)
- The Implicit Bias of Minima Stability: A View from Function Space (NeurIPS 2021 · similarity 0.92)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.91)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.