Research map / Optimization / Optimizers
SAMPa: Sharpness-aware Minimization Parallelized
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMPa, a parallelized variant of sharpness-aware minimization that computes the two SAM gradients simultaneously on two GPUs (with optimistic gradient correction), achieving ~2x per-iteration speedup and better generalization with a Lyapunov-based convergence proof for fixed perturbation radius.
Problem
SAM requires two sequentially dependent gradient computations per update, roughly doubling training time versus base optimizers like SGD, while many efficient variants either sacrifice accuracy or retain substantial overhead.
Approach
Replace the perturbation gradient at the clean weight with a gradient at an auxiliary sequence y_t (a look-ahead step from x_t), so that the perturbed-point gradient and the auxiliary gradient can be computed in parallel; interpolate with an optimistic gradient (OptGD) term via lambda, and prove convergence with a novel Lyapunov function even for fixed perturbation radius.
Key results
- On CIFAR-10 the average test accuracy rises from 95.81 (SAM, 2x temporal cost) to 95.99 for SAMPa-0.2 at half the sequential cost, and on CIFAR-100 from 78.90 to 79.22.
- SAMPa-0.2 reaches 10.94 s/epoch versus 18.81 s for SAM on ResNet-56 CIFAR-10 (including 7.5% communication overhead), second only to SAF at 10.09 s.
- On ImageNet-1K, Top-1 accuracy improves from 77.25 (SAM) to 77.44 for SAMPa-0.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the SAMPa-0.2 parallelized update instead of SAM | SAM · CIFAR-10 test (average, 5 models) | accuracy: 95.81_{\pm 0.15} -> 95.99_{\pm 0.08} | Helped |
| Add the SAMPa-0.2 parallelized update instead of SAM | SAM · CIFAR-100 test (average, 5 models) | accuracy: 78.90_{\pm 0.10} -> 79.22_{\pm 0.11} | Helped |
| Add the SAMPa-0.2 parallelized update instead of SAM | SAM · ImageNet-1K test | Top1 accuracy: 77.25_{\pm 0.05} -> 77.44_{\pm 0.03} | Helped |
| Add the SAMPa-0.2 parallelized update instead of SAM | SAM · CIFAR-100 test, ViT-B/16 fine-tuning | accuracy: 91.79_{\pm 0.12} -> 93.06_{\pm 0.16} | Helped |
| Add SAMPa-0.2 to the SAM variant | mSAM · CIFAR-10 test, ResNet-56 | accuracy: 94.28 -> 94.71 | Helped |
| Add SAMPa-0.2 to the SAM variant | ASAM · CIFAR-10 test, ResNet-56 | accuracy: 94.84 -> 94.95 | Helped |
| Set λ=0 rather than λ=0.2 (ablation) | SAMPa · CIFAR-10 test (average, 5 models) | accuracy: 95.99_{\pm 0.08} -> 95.86_{\pm 0.10} | Hurt |
| Set λ=0 rather than λ=0.2 (ablation) | SAMPa · CIFAR-100 test (average, 5 models) | accuracy: 79.22_{\pm 0.11} -> 78.91_{\pm 0.09} | Hurt |
| Use the previous perturbed-model gradient for the perturbation (OptSAM) | SAM · CIFAR-10 test, ResNet-56 | accuracy: 94.26 -> 93.99 | Hurt |
| Compute the perturbation and correction gradients on different batches (ablation) | SAM · CIFAR-10 test, ResNet-56 | accuracy: 94.26 -> 93.97 | Hurt |
| Use gradient penalization rather than the SAMPa-0.2 update (ablation) | SAMPa-0.2 · CIFAR-10 test, ResNet-32, 80% noisy labels | accuracy: 49.92 -> 48.26 | Hurt |
| Use SAMPa-0.2 rather than SAM | SAM · CIFAR-10, ResNet-56 | time per epoch (s): 18.81 -> 10.94 | Helped |
Surprising findings
- The naive OptSAM variant (reusing the previous perturbed gradient) performs worse than SGD on CIFAR-10 (e.g., 93.99 vs 94.20 on ResNet-56).
- Computing the two SAM gradients on different mini-batches (SAM-db) degrades accuracy below SGD, showing same-batch gradients are essential for SAM.
Limitations
- Requires two GPUs with equivalent memory to realize the speedup, so total compute resources match SAM.
- Convergence is proven only for lambda = 0 (the pure SAMPa variant), not for general lambda.
- Convergence analysis assumes convex and smooth objectives.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K GLUE CIFAR-10 with label noise
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K GLUE CIFAR-10 label noise
Metrics: test accuracy time per epoch (s) number of sequential gradients memory usage per GPU
Baselines: SGD SAM LookSAM AE-SAM SAF MESA ESAM RST mSAM ASAM SAM-ON VaSSO
Builds on: SAM optimistic gradient descent LookSAM ESAM MESA
Closest papers by meaning
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.91)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.90)
- Fundamental Convergence Analysis of Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.89)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.89)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.