Research map / Optimization / Nonconvex and smooth optimization
Fundamental Convergence Analysis of Sharpness-Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves fundamental convergence properties (gradient-to-zero, stationary accumulation points, function value and iterate convergence) for SAM and normalized variants under diminishing stepsizes, and for USAM/extragradient under a refined analysis, via inexact gradient descent frameworks.
Problem
SAM with constant stepsize does not provably converge (iterates, function values, or gradients), and prior SAM analyses gave only rate-type or neighborhood results rather than fundamental convergence properties.
Approach
Recasts SAM and its normalized variants (VaSSO, RSAM, F-SAM) as inexact gradient descent with absolute error bounded by the perturbation radius, and uses diminishing stepsizes and decaying/almost-constant perturbation radii to establish stationarity, gradient-to-zero, function-value and iterate convergence (with KL property); USAM and extragradient handled via relative-error IGD.
Key results
- On CIFAR-100, SAM with Diminish 3 stepsize reaches 79.70% with WRN28-10 versus 74.63% for constant stepsize.
- On CIFAR-10, Diminish 3 stepsize gives 94.75% (ResNet18) and 95.88% (WRN28-10), outperforming constant stepsize in the reported runs.
- Theorem 3.2 and Theorem 3.3 establish lim inf of gradient norm = 0 for convex SAM and gradient-to-zero plus stationary accumulation points for nonconvex normalized SAM under diminishing stepsizes and perturbation radii.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the constant stepsize with Diminish 3 (ablation) | SAM with ResNet18 · CIFAR-10 | accuracy: 94.10 ± 0.27 -> 94.75 ± 0.20 | Helped |
| replace the constant stepsize with Diminish 3 (ablation) | SAM with ResNet34 · CIFAR-10 | accuracy: 94.38 ± 0.47 -> 94.47 ± 0.08 | Helped |
| replace the constant stepsize with Diminish 3 (ablation) | SAM with WRN28-10 · CIFAR-10 | accuracy: 95.33 ± 0.23 -> 95.88 ± 0.10 | Helped |
| replace the constant stepsize with Diminish 3 (ablation) | SAM with ResNet18 · CIFAR-100 | accuracy: 71.77 ± 0.26 -> 75.65 ± 0.44 | Helped |
| replace the constant stepsize with Diminish 3 (ablation) | SAM with ResNet34 · CIFAR-100 | accuracy: 72.49 ± 0.23 -> 74.92 ± 0.76 | Helped |
| replace the constant stepsize with Diminish 3 (ablation) | SAM with WRN28-10 · CIFAR-100 | accuracy: 74.63 ± 0.84 -> 79.70 ± 0.12 | Helped |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with ResNet18 · CIFAR-10 | accuracy: 94.10 ± 0.27 -> 93.95 ± 0.34 | Hurt |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with ResNet34 · CIFAR-10 | accuracy: 94.38 ± 0.47 -> 93.94 ± 0.40 | Hurt |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with WRN28-10 · CIFAR-10 | accuracy: 95.33 ± 0.23 -> 95.18 ± 0.03 | Hurt |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with ResNet18 · CIFAR-100 | accuracy: 71.77 ± 0.26 -> 74.43 ± 0.12 | Helped |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with ResNet34 · CIFAR-100 | accuracy: 72.49 ± 0.23 -> 73.99 ± 0.70 | Helped |
| replace the constant stepsize with Diminish 1 (ablation) | SAM with WRN28-10 · CIFAR-100 | accuracy: 74.63 ± 0.84 -> 78.59 ± 0.03 | Helped |
Surprising findings
- Constant-stepsize SAM can fail to converge even on strongly convex quadratics, getting stuck at a nonoptimal cycle (Example 3.1).
- Diminishing stepsize outperformed the tuned constant stepsize across all tested architectures in the reported experiments.
Limitations
- Analysis restricted to deterministic settings; stochastic and random reshuffling variants left to future work.
- SAM combined with momentum methods not analyzed.
- Experiments limited to SAM on three image classification architectures on CIFAR datasets.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test accuracy training loss
Baselines: SAM with constant stepsize USAM VaSSO F-SAM RSAM
Builds on: Sharpness-Aware Minimization Inexact gradient descent (Khanh et al. 2023) Nonmonotone analysis of random reshuffling (Li et al. 2023) Unnormalized SAM (Andriushchenko and Flammarion 2022)
Closest papers by meaning
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.93)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.90)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.90)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- SAMPa: Sharpness-aware Minimization Parallelized (NeurIPS 2024 · similarity 0.89)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.