Research map / Optimization / Implicit bias and generalization

A Minimalist Example of Edge-of-Stability and Progressive Sharpening

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: Edge-of-stability dynamics

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes gradient descent on a width-one two-layer linear network with 2D input (one relevant, one irrelevant feature), proving progressive sharpening, edge-of-stability self-stabilization, and non-asymptotic loss decay along the full trajectory.

Problem

Explaining the Edge of Stability and Progressive Sharpening phenomena under large learning rates, where existing generalist analyses rely on hard-to-verify assumptions and minimalist examples only characterize asymptotic sharpness or use scalar inputs.

Approach

A two-layer width-one linear network with 2D Gaussian input where one feature has large scale but is irrelevant and the other small-scale feature determines the response; this yields non-asymptotic analysis of the entire GD trajectory, with the oscillation driven by the irrelevant direction and the loss decay driven by the relevant one.

Key results

• Proves progressive sharpening before EoS and self-stabilization during EoS for a width-one two-layer linear network, with sharpness bounded in [1.5/η, 4.71/η] after entering EoS.
• Shows GD converges to global minima with limiting sharpness at most (2+δ)/η, with loss decaying linearly at rate (1-2λ2/λ1)^2 in the constrained trajectory.
• Shows GFS sharpness lower bounded by λ1-1 for some initializations and that GD decreases GFS sharpness along its trajectory.

Measured effects reported in the paper

Surprising findings

• Increasing the learning rate in this minimalist setting neither speeds up nor slows down loss decrease, while a CIFAR-10 MLP experiment shows a larger learning rate (0.3 vs 0.1) slows training by about 50%.

Limitations

• Analysis of loss decay is restricted to the interval [0, T4] before the beta2 threshold is reached.
• Results are for a width-one linear network, leaving the connection to deep nonlinear networks empirical.
• Explanation of why larger learning rates slow training is only partial.

Experimental setup

Datasets: CIFAR-10

Metrics: sharpness training loss steps to reach loss threshold

Baselines: scalar network minimalist analyses generalist four-stage analyses gradient flow trajectory

Builds on: Edge of Stability (Cohen et al.) constrained trajectory / projected GD on stable set (Ahn et al.) gradient flow solution sharpness (Ma and Ying) minimalist scalar network analyses

Closest papers by meaning

• Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability (NeurIPS 2022 · similarity 0.92)
• Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More (ICML 2025 · similarity 0.91)
• Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.89)
• A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
• Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)