Research map / Optimization / Implicit bias and generalization
Flat Channels to Infinity in Neural Loss Landscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies "channels to infinity" in neural loss landscapes, flat valleys where two neurons' output weights diverge with converging input weights, which implement gated linear units in the limit.
Problem
Understanding the structure of neural network loss landscapes, specifically flat regions connected to symmetry-induced saddle lines, and where gradient-based training converges in them.
Approach
Shows that gradient flow trajectories near neuron-duplication saddle lines enter channels where two neurons' output weights diverge to ±infinity while their input weights converge; in the limit the pair implements a gated linear unit (a finite-difference approximation of a directional derivative of the activation).
Key results
- Channels to infinity are reached with a substantial probability (roughly 10% to 30% reported for plateau saddles in the 2D setup) across random Glorot initializations, with more channels for rougher GP targets.
- In a 2-neuron erf toy example, the global optimum lies at a minimum at infinity (channel 2), where the pair of neurons implements a gated linear unit.
- Along channels, loss decreases as epsilon^2 and sharpness diverges as 1/epsilon^2; SGD and ADAM get stuck at the edge of stability near channel entrances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add bias terms to neurons (ablation) | Single-hidden-layer MLPs with neuron-duplication saddle lines · Modified 2D Rosenbrock regression | frequency of plateau saddles | Hurt |
| Increase the Gaussian-process kernel scale (ablation) | MLPs trained on Gaussian-process regression datasets · Gaussian-process datasets | probability of reaching a channel | Helped |
| Use SGD or ADAM instead of gradient flow (ablation) | Two-neuron erf network · One-dimensional toy regression | progress along the channel | Hurt |
| Use a finite step size (ablation) | Gradient descent along a channel to infinity · One-dimensional toy regression | maximum Hessian eigenvalue: 2/η | Hurt |
| Apply the jump procedure to decrease ε along the channel (ablation) | Neural networks converging to channels to infinity · Rosenbrock regression | loss | Helped |
| Replace softplus neurons with a pair whose input weights converge and output weights diverge with opposed signs | Fully connected layer · Gated-linear-unit functional limit | ability to implement a gated linear unit | Helped |
| Use smooth activation functions (softplus, erf, or sigmoid) (ablation) | MLPs trained on regression settings · Regression datasets | occurrence of channels to infinity | No clear effect |
| Use ReLU activation (ablation) | MLPs · Infinite-data regression | existence of channels to infinity | No clear effect |
Surprising findings
- Adding bias terms drastically reduces the number of stable plateau saddles in neuron-duplication saddle lines.
- Channel minima are sharper along one direction yet extremely flat along the saddle-line direction, and SGD/ADAM appear to converge to what look like flat finite-norm minima.
Limitations
- Theory developed for single-hidden-layer MLPs with smooth activations and MSE loss; generalization to large-scale models is untested and left as ongoing work.
- Finding the saddle line parameterization of a given channel is an open challenge with a heuristic that is not guaranteed to work.
- Channels reached with ODE gradient flow; with reasonable compute budgets 1/epsilon never exceeded a few hundred, so minimum at infinity is supported empirically rather than proven.
Experimental setup
Datasets: modified Rosenbrock function Gaussian process datasets (Matern32, s in {0.1, 0.5, 2, 10})
Benchmarks: modified Rosenbrock regression Gaussian process regression (s=0.1, 0.5, 2, 10) erf toy target
Metrics: training loss convergence probability to channels Hessian eigenvalues gradient norm cosine distance of input weights
Baselines: SGD ADAM
Builds on: Fukumizu and Amari neuron splitting gradient flow edge of stability KenCarp58
Closest papers by meaning
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.89)
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures (NeurIPS 2025 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.88)
- Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality (ICLR 2025 · similarity 0.88)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.