Research map / Optimization / Implicit bias and generalization
Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes a 2-layer linear "UV model" trained on one example via fixed-point and vector-field analysis to explain sharpness reduction, progressive sharpening, edge of stability, and a period-doubling route to chaos, then checks predictions on real CNN/FCN/ResNet settings.
Problem
Explaining the robust sharpness (top Hessian eigenvalue) phenomena during gradient descent training of neural networks: early sharpness reduction, progressive sharpening, edge of stability, and the dependence on initialization and parameterization.
Approach
Reduce GD dynamics of a 2-layer linear network on a single example to a closed 2D map in function space (residual Δf and trace-of-Hessian λ), then use fixed-point and vector-field analysis to explain sharpness trends, the EoS condition, and period-doubling to chaos, with the parameterization entering via effective width.
Key results
- The UV model reproduces the four training regimes (early transient, intermediate saturation, progressive sharpening, EoS) and predicts EoS requires eta > sqrt(n_eff)/(||x|| y).
- Real-world experiments confirm that SP with large initialization and wide networks do not show EoS, while muP and small-initialization networks do.
- Models trained on random synthetic data show period-doubling route to chaos at EoS, whereas CIFAR-10-trained models show dense sharpness bands and long-range temporal correlations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SP with large initialization instead of SP with small initialization | ReLU FCN · CIFAR-10, 5,000-example subset | edge-of-stability behavior | Hurt |
| Use μP instead of SP with large initialization | ReLU FCN · CIFAR-10, 5,000-example subset | early sharpness reduction | Hurt |
| Use μP instead of SP with large initialization | ReLU FCN · CIFAR-10, 5,000-example subset | edge-of-stability behavior | Helped |
| Train on random synthetic data instead of CIFAR-10 | 2-layer linear FCN · Random synthetic dataset vs CIFAR-10, 5,000 examples | period-doubling route to chaos | Mixed |
| Use small initial weight variance | Real-world models · Edge of stability phase diagram | edge-of-stability behavior | Helped |
| Increase interpolating parameter s | Real-world models · Edge of stability phase diagram | edge-of-stability behavior | Helped |
| Increase learning-rate constant c | Real-world models · Edge of stability phase diagram | edge-of-stability behavior | Helped |
| Observe EoS | Real-world models · Test accuracy heatmaps | test accuracy | Helped |
| Use a random dataset with power-law singular value spectrum in the input | 2-layer FCN · Synthetic dataset | period-doubling route to chaos | Hurt |
Surprising findings
- Standard parameterization with large initialization does not exhibit edge of stability even when loss drops below 1e-5.
- The UV model under NTP at considerable widths does not display EoS, in contrast to prior analyses of 2-layer linear networks.
Limitations
- UV model does not capture non-monotonic loss decrease at EoS and loss catapult is absent for n_eff=1.
- UV model does not capture long-range correlations in sharpness seen in real datasets.
- Closure of dynamics to few variables is difficult for non-linear DNNs, limiting the fixed-point method.
Experimental setup
Datasets: CIFAR-10 MNIST Fashion-MNIST Wikitext-2 random synthetic dataset teacher-student dataset random power-law dataset generative image dataset
Benchmarks: CIFAR-10 MNIST Fashion-MNIST
Metrics: top Hessian eigenvalue (sharpness) training loss test accuracy normalized sharpness eta*lambda/2 power spectrum of sharpness
Baselines: Standard Parameterization Neural Tangent Parameterization muP
Builds on: Edge of Stability (Cohen et al.) catapult dynamics (Lewkowycz et al.) maximal update parameterization (muP) Neural Tangent Parameterization
Closest papers by meaning
- Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More (ICML 2025 · similarity 0.93)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.90)
- A Minimalist Example of Edge-of-Stability and Progressive Sharpening (NeurIPS 2025 · similarity 0.89)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.88)
- Super Consistency of Neural Network Landscapes and Learning Rate Transfer (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.