Research map / Optimization / Implicit bias and generalization
Improving Sharpness-Aware Minimization by Lookahead
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds an extra-gradient-style lookahead to Sharpness-Aware Minimization (SAM), yielding Lookahead-SAM, Opt-SAM and AO-SAM variants that avoid saddle points and reach flatter minima at SAM-level convergence rates.
Problem
SAM can suffer from convergence instability, oscillating around saddle points, which leads to slow convergence and inferior generalization.
Approach
Applies lookahead/extra-gradient and optimistic-gradient ideas to SAM's minimax formulation, simplifying the extrapolation so the perturbation is reduced near saddle points; the optimistic variant reuses past gradients and the adaptive variant (AO-SAM) skips SAM steps using AE-SAM's statistical test.
Key results
- On CIFAR-100 with ResNet-18, Opt-SAM reaches 80.76% vs SAM 80.17% at the same 100% SAM updates.
- AO-SAM achieves 96.82% on CIFAR-10 with ResNet-18 using 61.1% SAM updates, versus SAM's 96.52% at 100%.
- On ImageNet with ResNet-50, AO-SAM reaches 77.68% vs SAM 77.47% while using 61.1% SAM updates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a lookahead step to SAM | SAM · CIFAR-10, ResNet-18 | accuracy: 96.52 ±0.12 -> 96.81 ±0.01 | Helped |
| Reuse the previous gradient in the lookahead step | SAM · CIFAR-100, ResNet-18 | accuracy: 80.17 ±0.05 -> 80.76 ±0.15 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · CIFAR-10, ResNet-18 | accuracy: 96.52 ±0.12 -> 96.82 ±0.04 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · CIFAR-100, ResNet-18 | %SAM: 100.0 ±0.0 -> 61.2 ±0.0 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · CIFAR-100, ResNet-18 | accuracy: 80.17 ±0.05 -> 80.70 ±0.14 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · MRPC | Accuracy: 87.9 -> 88.7 | Helped |
| Reuse the previous gradient in the lookahead step | SAM · MRPC | F1-score: 91.4 -> 92.2 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · ImageNet, ResNet-50 | accuracy: 77.47 ± 0.12 -> 77.68 ± 0.04 | Helped |
| Use adaptive SAM updates with optimistic lookahead | SAM · CIFAR-10, ResNet-18 | training time (seconds): 6,780 -> 4,704 | Helped |
| Use direct extra-gradient updates instead of SAM | SAM · CIFAR-100, ResNet-18 | accuracy: 80.17 ±0.05 -> 79.91 ±0.16 | Hurt |
| Use optimistic-gradient updates without the proposed lookahead-SAM construction | SAM · CIFAR-100, ResNet-18 | accuracy: 80.17 ±0.05 -> 79.92 ±0.08 | Hurt |
Surprising findings
- Direct adaptation of extra-gradient and optimistic gradient to SAM gives only marginal gains and OG is slower than SAM on CIFAR-100.
Limitations
- Lookahead-SAM requires two gradient computations per iteration, making it slower than SAM.
- Theoretical convergence analysis does not assume convexity but the guarantees are for stationary points at the SAM rate only.
- Evaluated on standard image and one NLP benchmark; distribution shift scenarios left for future work.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet MRPC
Benchmarks: CIFAR-10 CIFAR-100 ImageNet MRPC
Metrics: testing accuracy %SAM (fraction of SAM updates) training time GPU memory Hessian eigenvalue (lambda1, lambda1/lambda5) F1-score
Baselines: SAM ESAM ASAM GSAM SS-SAM AE-SAM ERM EG OG
Builds on: Sharpness-Aware Minimization Extra-gradient Optimistic gradient AE-SAM
Closest papers by meaning
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.91)
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.91)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.90)
- Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.90)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.90)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.