Research map / Optimization / Implicit bias and generalization
Sharpness-Aware Minimization: General Analysis and Improved Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Unified SAM, a convex combination of normalized SAM and USAM controlled by lambda, and proves convergence under the Expected Residual condition for PL and non-convex functions under arbitrary sampling.
Problem
Convergence of stochastic SAM variants (normalized SAM and USAM) is not well understood: analyses rely on bounded variance/gradient assumptions, the benefit of normalization is unclear, and guarantees under non-uniform or general sampling are missing.
Approach
A single Unified SAM update that interpolates between normalized SAM (lambda=0) and USAM (lambda=1) via a convex combination of the normalization factors, analyzed under the Expected Residual condition and arbitrary sampling so that bounded variance is not needed.
Key results
- Deterministic experiments show normalized SAM (lambda=0) converges to the exact solution while lambda>0 converges only to a neighborhood, growing with lambda.
- On CIFAR-10 with WRN-28-10, Unified SAM with lambda_t=1-1/t reaches 96.34% (ResNet-18) and 97.06% (WRN-28-10) vs SAM-like lambda=0 at 96.12% and 96.77%.
- On CIFAR-100 with WRN-28-10, lambda_t=1-1/t attains 83.66% vs 83.07% for lambda=0 (ResNet-18 best is 80.01% at lambda=0.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use λ=1−1/t instead of λ=0.0 (ablation) | Unified SAM, WRN-28-10 · CIFAR-10 test, ρ=0.2 | test accuracy (%): 95.8 -> 95.98 | Helped |
| Use λ=1−1/t instead of λ=0.0 (ablation) | Unified SAM, WRN-28-10 · CIFAR-100 test, ρ=0.3 | test accuracy (%): 80.94 -> 81.71 | Helped |
| Vary λ across tested values (ablation) | Unified SAM, WRN-28-10 · CIFAR-10 test | test accuracy (%): 95.7 -> 95.9; 95.8 -> 95.98; 95.35 -> 95.99; 95.46 -> 95.79 | Mixed |
| Vary λ across tested values (ablation) | Unified SAM, WRN-28-10 · CIFAR-100 test | test accuracy (%): 80.84 -> 81.01; 81.12 -> 81.53; 80.94 -> 81.71; 80.1 -> 81.70 | Mixed |
| Use λ=1−1/t instead of λ=1.0 (ablation) | Unified VaSSO, WRN-28-10 · CIFAR-10 test | test accuracy (%): 97.03 -> 97.06 | Helped |
| Use λ=1−1/t instead of λ=1.0 (ablation) | Unified VaSSO, WRN-28-10 · CIFAR-100 test | test accuracy (%): 83.51 -> 83.66 | Helped |
| Use decreasing rather than constant step sizes (ablation) | Unified SAM · Logistic regression, synthetic data | accuracy | Helped |
| Use importance sampling rather than uniform single-element sampling (ablation) | Unified SAM · Ridge regression, synthetic data | convergence | Helped |
| Set λ=0 rather than λ>0 (ablation) | Deterministic Unified SAM · Ridge regression, synthetic data | objective convergence | Helped |
Surprising findings
- Deterministic USAM (lambda=1) does not converge to the exact solution on PL problems, only to a neighborhood.
- Intermediate lambda values, not the endpoints, gave the best accuracy in most image classification settings.
Limitations
- Optimal lambda is not characterized theoretically and must be tuned empirically.
- Step sizes are constant-tuned for theory; the analysis does not cover adaptive step sizes rho_t and gamma_t.
- Experiments are limited to small image classification benchmarks (CIFAR) and synthetic problems.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 synthetic ridge regression synthetic logistic regression
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test accuracy distance/suboptimality convergence on synthetic problems
Baselines: SGD SAM USAM VaSSO
Builds on: SAM USAM SGD Expected Residual condition
Closest papers by meaning
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.93)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.93)
- Fundamental Convergence Analysis of Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.93)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.92)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.92)
- SAMPa: Sharpness-aware Minimization Parallelized (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.