Research map / Optimization / Implicit bias and generalization

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding

ICLR 2025 · Optimization › Implicit bias and generalization · approach: Shallow ReLU network convergence · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes directional stationary points of the empirical squared loss for one-hidden-layer ReLU-like networks, showing that stationary points without escape neurons are local minima and analyzing saddle escape and unit-replication network embedding.

Problem

Loss landscape of shallow ReLU-like networks is non-differentiable, so standard stationarity notions are inadequate for characterizing where gradient descent stalls, saddle escape in vanishing-initialization saddle-to-saddle dynamics, and how network embedding reshapes stationary points.

Approach

Defines stationarity via one-sided directional derivatives (directional stationarity) and derives closed-form ODDs along radial and tangential input-weight directions; shows stationary points without escape neurons are local minima (and for scalar output, escape neurons imply non-minimality), and uses this to refine saddle escape and embedding results.

Key results

• Stationary points without escape neurons are local minima; for scalar output the presence of an escape neuron implies the stationary point is not a local minimum.
• In a 50-neuron ReLU network trained from vanishing initialization on 5 samples, the loss shows three plateaus; the last corresponds to a local minimum, and the two saddle escapes occur near epochs 42k and 152k.
• Unit replication preserves stationarity iff all tangential derivatives of the replicated neuron vanish or all output-scaling factors gamma_l are non-negative.

Surprising findings

• Directional stationarity can fail to capture GD stalling on measure-zero sets, e.g., a point with negative one-sided derivative can still stall GD for certain starting points.
• Unit replication can preserve type-1 local minimality, contradicting the expectation from a prior theorem that embedding a local minimum yields a strict saddle; the prior theorem's assumptions do not apply here.

Limitations

• Analysis restricted to one-hidden-layer networks; deeper networks are only sketched.
• Type-2 local minima (with escape neurons, scalar-output excluded) are argued to be rare but not fully characterized for multiple outputs.
• Directional stationarity may fail to capture GD stalling on probability-zero sets.

Experimental setup

Datasets: synthetic 1-D inputs with bias term (5 samples)

Metrics: loss value stationarity conditions (ODDs)

Builds on: Clarke stationarity saddle-to-saddle dynamics embedding principle

Closest papers by meaning

• Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape (ICLR 2026 · similarity 0.92)
• Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.90)
• Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs (NeurIPS 2023 · similarity 0.90)
• Flat Channels to Infinity in Neural Loss Landscapes (NeurIPS 2025 · similarity 0.89)
• Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.89)
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.88)