Research map / Optimization / Implicit bias and generalization
Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies progressive sharpening and edge of stability with a deep linear network having one neuron per layer, deriving sharpness bounds in terms of a dataset difficulty measure and analyzing how depth, data, and SGD batch size and step size affect sharpness.
Problem
Progressive sharpening (sharpness rising during gradient descent until the edge of stability) is commonly observed, but its mechanisms and dependence on dataset, depth, and optimizer are poorly understood.
Approach
Uses a minimalist deep linear network with a single neuron per layer and defines a dataset difficulty Q = sum d_i^2/sigma_i^2; proves sharpness bounds at minimizers and at convergence under gradient flow, and shows that SGD's stochasticity increases layer imbalance more than GD, reducing progressive sharpening.
Key results
- Sharpness at convergence under gradient flow is well predicted by the predicted sharpness S_D = (sigma_1^2/N) Q^{(D-1)/D} across CIFAR10, SVHN, and speech datasets.
- Theory predicts depth increases sharpness when Q>1 and decreases it when Q<1; Q for CIFAR10 2-label subsets grows from 0.22 at N=100 to 44.44 at N=1000.
- A single SGD step increases layer imbalance more than a GD step, with the effect larger for smaller batch size and larger step size, consistent with less progressive sharpening under SGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase training dataset size | Gradient flow training · CIFAR10, SVHN, and Google speech 2-label subsets | progressive sharpening | Helped |
| increase network depth | Fully connected networks with fixed width, trained with gradient flow · Squared-loss training | progressive sharpening | Helped |
| increase network depth | Minimalist model · Datasets with Q>1 | sharpness | Helped |
| increase network depth | Minimalist model · Datasets with Q<1 | sharpness | Hurt |
| increase SGD batch size | SGD · Mean squared loss training | progressive sharpening | Helped |
| increase SGD learning rate | SGD · Mean squared loss training, especially with small batches | progressive sharpening | Hurt |
| use SGD stochasticity rather than a full batch | Minimalist two-layer model · Training dynamics | progressive sharpening | Hurt |
| decrease SGD batch size | Minimalist two-layer model · Training dynamics | sharpness | Hurt |
| increase SGD step size | Minimalist two-layer model · Training dynamics | sharpness | Hurt |
| increase dataset difficulty Q | Balanced minimalist model · Sharpness at a global minimizer | sharpness | Helped |
Surprising findings
- Wider networks showed less progressive sharpening with cross-entropy loss in prior work, but this trend did not hold for squared loss, so network width was excluded as a key factor.
Limitations
- Theoretical results are for a single-neuron-per-layer linear model; extension to nonlinear networks is only preliminary.
- Theorem on SGD does not fully prove that SGD yields larger layer imbalance and smaller final sharpness at convergence.
- Bounds for deep networks assume balanced layers and gradient flow convergence.
Experimental setup
Datasets: CIFAR10 SVHN Google speech commands
Benchmarks: CIFAR10 2-label subsets SVHN Google speech commands
Metrics: sharpness (top Hessian eigenvalue) predicted sharpness vs empirical sharpness dataset difficulty Q layer imbalance C
Baselines: gradient flow gradient descent stochastic gradient descent
Builds on: edge of stability self-stabilization central flow progressive sharpening
Closest papers by meaning
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.93)
- A Minimalist Example of Edge-of-Stability and Progressive Sharpening (NeurIPS 2025 · similarity 0.91)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.