Research map / Optimization / Implicit bias and generalization

Simplicity Bias via Global Convergence of Sharpness Minimization

ICML 2024 · Optimization › Implicit bias and generalization · approach: Stability-based generalization analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that label noise SGD (and 1-SAM) on two-layer networks with certain activations globally converges to the minimizer of trace-of-Hessian on the zero-loss manifold, which yields a rank-one feature matrix (simplicity bias).

Problem

Whether sharpness-reduction implicit bias of label noise SGD provably yields simplicity bias (low-rank features) in non-linear two-layer networks, and whether the Riemannian gradient flow of trace-of-Hessian converges globally to its minimizers.

Approach

Proves that for two-layer networks with fixed second layer and activations with positive, convex phi' and beta-normality, the trace of Hessian on the zero-loss manifold has a unique first-order optimum (all neurons collapse to one feature vector in high dimension), and that its Riemannian gradient flow converges linearly from any initialization via local geodesic convexity near approximate station

Key results

• Under the assumptions, the trace of Hessian has a unique set of first-order optima in which all neurons' pre-activations on each training point equal phi^{-1}(y_i/m), so the feature matrix is rank one in the high-dimensional regime d > n.
• The Riemannian gradient flow of trace of Hessian reaches epsilon-stationarity after time O(log(1/epsilon)) with linear convergence of gradient norm squared for all initializations.
• Label noise SGD with sufficiently small step size converges in probability to the global minimizer of trace of Hessian on the zero-loss manifold, with explicit constants depending only on the activation and data coherence.

Measured effects reported in the paper

Surprising findings

• In low dimensions (d < n), where the feature matrix cannot be rank one, neurons still empirically cluster into a small set of vectors, beyond the theory's regime.

Limitations

• Theory applies only to activations with strictly positive and convex phi' (plus beta-normality), with cube activation as a special case, excluding e.g. ReLU.
• Analysis requires high-dimensional inputs (d at least n, linearly independent data); low-dimensional case is only observed empirically.
• Results are for two-layer networks with fixed unit second-layer weights and extension to deeper networks is conjectured.

Experimental setup

Datasets: synthetic data (uniform entries on [0,1])

Metrics: eigenvalues of feature matrix rank of feature matrix neuron feature collapse

Builds on: label noise SGD implicit bias (Blanc et al. / Li et al.) sharpness-aware minimization low-rank feature bias conjecture

Closest papers by meaning

• Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.89)
• Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.89)
• Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.89)
• A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
• Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.89)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)