Research map / Optimization / Implicit bias and generalization

Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: Edge-of-stability dynamics · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that the (stochastic) gradient descent map is non-singular for almost all step-sizes for neural networks (FC, convolutional, softmax attention) with piecewise analytic activations, by showing the loss is analytic almost everywhere.

Problem

Prior results on GD avoiding saddles and on minima stability assume the GD map is non-singular, but this was unjustified for neural networks with piecewise analytic activations (e.g., ReLU) and Lipschitz-smooth assumptions often fail.

Approach

Shows the network map and empirical loss are analytic almost everywhere using a layered analogue of the chain rule, then proves GD maps are non-singular for almost all step-sizes via a Jacobian-determinant argument on analytic functions (eigenvalues of the Hessian and zero sets of analytic functions).

Key results

• Proves that for almost all step-sizes eta, the GD and SGD maps of neural networks with piecewise analytic activations (FC, convolutional, softmax attention) are non-singular.
• Shows the empirical loss is analytic almost everywhere, so the boundary and non-analytic sets have measure zero.
• Illustrative 2-layer ReLU experiment shows periodic orbits and step-size-dependent stability of minima for GD vs SGD.

Surprising findings

• Non-singularity of the GD map can fail at specific step-sizes eta = 1/lambda for Hessian eigenvalues, so the result holds only for almost all step-sizes.

Limitations

• Recurrent networks are not covered; the authors only conjecture non-singularity for them.
• Overparameterized case (n_theta > m) requires more care for showing saddles/unstable minima lie in measure-zero sets.
• Only supervised learning with fixed analytic loss considered; other optimizers (mirror descent, proximal) not proven.

Experimental setup

Builds on: Lee et al. gradient descent converges to minimizers (saddle avoidance) Rademacher's theorem Lyapunov-exponent stability of minima for GD and SGD

Closest papers by meaning

• Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.88)
• Block Coordinate Descent for Neural Networks Provably Finds Global Minima (NeurIPS 2025 · similarity 0.87)
• Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.86)
• The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.86)
• On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.86)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.86)