Research map / Optimization / Implicit bias and generalization
Topological obstruction to the training of shallow ReLU neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient flow on a two-layer homogeneous (ReLU) network with scalar output confines training to a product of hyperquadrics whose number of connected components can be exponential in the number of "pathological" neurons, and that permutation plus rescaling reduces this to a linear count of effective components.
Problem
Gradient flow training of shallow ReLU networks is constrained by rescaling symmetry, which may make the global optimum unreachable from some initializations; the topology of the reachable parameter set is unclear.
Approach
Rescaling invariance makes each neuron's quantity ||W1_k||^2 - ||W2_k||^2 conserved under gradient flow, so trajectories lie on an invariant set factored into per-neuron hyperquadrics; computing its Betti numbers shows disconnection when the output is scalar and some neurons start with negative conserved quantity, and permutations merge components into a count linear in the number of such neurons.
Key results
- For e=1 and d>1, the invariant set has 2^{l_-} connected components, while the number of effective components under permutations and rescalings is 1 + l_-.
- In the toy experiment with c=(-0.1,-0.1), trajectories stay confined to the positive region and fail to approximate F, whereas c=(-0.1,+0.1) lets the network flip the sign of the second output weight and learn F.
- On the breast cancer experiment, average test BCE loss decreases as the number of non-pathological neurons l_+ increases over 100 random initializations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize with one non-pathological neuron (c=(-0.1, +0.1)) instead of two pathological neurons (c=(-0.1, -0.1)) | Two-neuron shallow ReLU network · Toy scalar regression | approximation of F | Helped |
| Increase the number of non-pathological neurons | Two-layer ReLU network · Breast cancer | test BCE loss | Helped |
| Initialize with pathological neurons whose outgoing weights must retain their signs | Shallow ReLU network trained with gradient flow · Scalar-output tasks with d>1 | global optimum reachability | Hurt |
Surprising findings
- Under common initializations (Xavier, Kaiming), the probability of pathological neurons becomes negligible for high input dimension and width, so the obstruction is unlikely in practice.
- Connected components scale exponentially in the number of pathological neurons, but effective components under permutation scale only linearly.
Limitations
- Analysis restricted to a single hidden layer; multi-layer invariant sets do not factor into products of hyperquadrics.
- Results are for gradient flow; finite step size, SGD, and Adam break the conservation laws.
- Regularization such as l_p penalties breaks rescaling invariance.
Experimental setup
Datasets: Synthetic toy regression data (F(x1,x2)=-(x1+x2), 8000 points) breast cancer dataset
Benchmarks: breast cancer dataset (BCE test loss)
Metrics: MSE loss BCE test loss Betti numbers / number of connected components
Builds on: Neuron rescaling symmetry of ReLU networks Künneth formula Conservation laws for gradient flow in homogeneous networks
Closest papers by meaning
- Topology and geometry of the learning space of ReLU networks: connectivity and singularities (ICLR 2026 · similarity 0.90)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.90)
- Bounds on Over-Parameterization for Guaranteed Existence of Descent Paths in Shallow ReLU Networks (ICLR 2020 · similarity 0.90)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.87)
- The Symmetries of Three-Layer ReLU Networks (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.