Research map / Optimization / Implicit bias and generalization
Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that applying SAM only in the last few epochs after SGD training yields generalization and sharpness comparable to full SAM, and gives a linear-stability and convergence theory explaining a two-phase escape-then-converge dynamic, extended to adversarial training.
Problem
Understanding the implicit bias of SAM toward flatter minima relative to SGD, and when during training SAM's effect on generalization and sharpness takes hold, given that existing theory relies on unrealistic assumptions.
Approach
Using a switching method (SGD-to-SAM and SAM-to-SGD at varying epochs), the paper finds that SAM's benefits come from the late phase of training; it explains this with a two-phase picture (exponential escape from SGD's sharp minimum, then fast convergence to a flatter minimum within the same valley) backed by a linear stability analysis under gradient-noise alignment and a sub-quadratic landscape
Key results
- Applying SAM for only the last 1-5 epochs of WideResNet-16-8 on CIFAR-10 yields test error and loss comparable to full SAM training (over 5 seeds).
- Sharpness (Hessian spectral norm) of SGD-to-SAM solutions drops sharply once the SAM proportion exceeds zero, tracking full SAM.
- Switching late to adversarial training recovers robust error comparable to full AT, while natural error only slightly rises.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from SGD to SAM for the final 1–5 epochs (ablation) | WideResNet-16-8 · CIFAR-10 | test error | Helped |
| switch from SGD to SAM late in training (ablation) | image classifiers · CIFAR-10 and CIFAR-100 | test error | Helped |
| switch from SGD to SAM late in training (ablation) | image classifiers · CIFAR-10 and CIFAR-100 | Hessian spectral norm | Helped |
| use SAM early, then switch to SGD (ablation) | image classifiers · CIFAR-10 and CIFAR-100 | test error | Hurt |
| use SAM early, then switch to SGD (ablation) | image classifiers · CIFAR-10 and CIFAR-100 | Hessian spectral norm | Hurt |
| vary SAM perturbation radius before switching to SGD (ablation) | SAM→SGD · CIFAR-10 | test error | No clear effect |
| increase SAM perturbation radius during full SAM training (ablation) | SAM · CIFAR-10 | test error | Helped |
| switch from SGD to adversarial training late in training (ablation) | image classifiers · adversarial examples | robust error | Helped |
| switch from SGD to adversarial training late in training (ablation) | image classifiers · clean examples | natural error | Mixed |
| switch from SGD to ASAM late in training (ablation) | image classifiers · various architectures and datasets | test error | Helped |
| switch from SGD to USAM late in training (ablation) | image classifiers · various architectures and datasets | test error | Helped |
Surprising findings
- Early-phase SAM has limited effect: SAM-to-SGD models retain a substantial generalization gap versus full SAM even when SAM covers 80% of training.
- Sharpness of SAM-to-SGD solutions rapidly rises to the edge of stability (about 2/eta) after switching to SGD, regardless of how late the switch occurs.
Limitations
- Theory relies on simplifications such as constant perturbation radius, squared loss, linearized model and independent inner/outer noise.
- Theory only explains late-phase SAM behavior and does not account for early-phase SAM effects.
- The sub-quadratic landscape proposition is analyzed for p=1 and full-batch SAM.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test error test loss Hessian spectral norm sharpness robust error natural error
Baselines: SGD SAM ASAM USAM adversarial training
Builds on: Sharpness-Aware Minimization adversarial training Edge of Stability USAM
Closest papers by meaning
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.93)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.93)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.92)
- Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.91)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.91)
- Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.