Research map / Optimization / Implicit bias and generalization

SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that gradient flows of SAD (sublinear analytic definable) neural networks either converge to a critical point or diverge to infinity with loss tending to an asymptotic critical value, and that for polynomial targets with large enough architecture and data the loss has no minimum (infimum zero), so sufficiently good initializations diverge.

Problem

Understanding when gradient flow on neural network loss landscapes converges to a good minimum versus diverges to infinity, given that parameters may leave bounded regions.

Approach

Uses o-minimal structure theory (definability, Kurdyka-Lojasiewicz inequality, uniform finiteness, definable dimension theory) to show a dichotomy for definable gradient flows and defines SAD activations (sublinear, analytic, definable) under which polynomial targets cannot be fit exactly, so the infimum is zero but unattained and gradient flows diverge.

Key results

• Gradient flow for definable C^1 neural network losses either converges to a critical point or diverges to infinity with loss converging to a generalized critical value; a threshold epsilon exists so flows starting below inf + epsilon converge to the infimum.
• For polynomial targets of degree at least 2 with sufficiently large architecture and data, the loss has no global minimum but infimum zero, so optimal parameters are only achieved asymptotically.
• Numerical experiments (GD and Adam on polynomial targets, deep Kolmogorov PDEs, MNIST) consistently show parameter norm growth as loss decreases, growing roughly logarithmically under GD.

Surprising findings

• Divergence of parameters occurs even without the sufficiently-good initialization required by the theorem, in practice.
• The parameter norm grows logarithmically under gradient descent in experiments rather than as the O(sqrt(t)) bound suggests.

Limitations

• Results are proven for gradient flow only; the authors cannot yet prove analogous results for SGD or gradient descent.
• Activations must be C^1 or analytic, so ReLU is excluded; the divergence result fails for ReLU where global minima exist for shallow networks.
• Numerical experiments are small-scale proofs of concept with limited datasets.

Experimental setup

Datasets: MNIST

Benchmarks: MNIST

Metrics: mean squared error loss parameter norm relative mean squared error

Baselines: gradient descent Adam

Builds on: Kurdyka-Lojasiewicz inequality Pfaffian functions deep Kolmogorov method Adam

Closest papers by meaning

• The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks (ICLR 2023 · similarity 0.89)
• Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality (ICLR 2025 · similarity 0.88)
• Flat Channels to Infinity in Neural Loss Landscapes (NeurIPS 2025 · similarity 0.88)
• Convergence of the Gradient Flow for Shallow ReLU Networks on Weakly Interacting Data (NeurIPS 2025 · similarity 0.88)
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
• Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.87)