Research map / Architectures and efficiency / RNNs and normalization
Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AUSAM, which probabilistically samples a subset of each mini-batch using a gradient-norm proxy (loss difference before/after SAM perturbation, averaged over history) to speed up Sharpness-Aware Minimization while keeping its accuracy.
Problem
SAM requires two forward and backward passes per update, roughly doubling training cost over the base optimizer; existing acceleration methods often degrade generalization.
Approach
Select high-gradient-norm samples within each mini-batch for SAM updates, approximating per-sample gradient norms cheaply from the loss difference before and after SAM perturbation (DLP), averaged over epochs, so the sampled subset gives an asymptotically unbiased estimate of the full-batch gradient.
Key results
- On CIFAR-100 with ResNet-18, AUSAM-0.5 reaches 80.67% accuracy at 132% of SAM speed versus SAM's 81.04% at 100%, per the paper's table.
- On CIFAR-100 with WideResNet-28-10, AUSAM-0.6 reaches 84.73% accuracy vs. SAM's 84.71%, at 152% relative speed.
- AUSAM-0.5 is reported as roughly 70% faster than SAM while achieving comparable accuracy on CIFAR-10/100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AUSAM sampling rather than full-batch SAM | ResNet-18 · CIFAR-10 | SAM%: 100% -> 123% | Helped |
| Use AUSAM sampling rather than full-batch SAM | WideResNet-28-10 · CIFAR-100 | accuracy: 84.71 -> 84.73 | Helped |
| Use AUSAM sampling rather than full-batch SAM | ResNet-18 · Tiny-ImageNet | accuracy: 64.43 -> 64.49 | Helped |
| Use AUSAM sampling rather than full-batch SAM | ResNet-50 · Tiny-ImageNet | accuracy: 67.69 -> 66.65 | Hurt |
| Use AUSAM sampling rather than full-batch SAM | MobileNetv1 · Tiny-ImageNet | accuracy: 58.49 -> 58.94 | Helped |
| Use AUSAM sampling rather than SAM | SimCC with Adam · MP II | Mean: 89.4 -> 89.5 | Helped |
| Use AUSAM sampling rather than SAM | ResNet-18 · CIFAR-10 QAT | accuracy: 89.75 -> 89.63 | Hurt |
| Use AUSAM sampling rather than SAM | MobileNet · CIFAR-10 QAT | accuracy: 84.72 -> 84.91 | Helped |
| Apply InfoBatch during either SAM loss calculation | SAM · CIFAR-100 | accuracy: 81.24 -> 79.68; 81.24 -> 79.57 | Hurt |
| Set sampling rate α to 0.4 rather than 0.6 (ablation) | AUSAM on ResNet-18 · CIFAR-100 | accuracy: 80.69 -> 80.14 | Hurt |
| Set sampling rate α to 0.5 rather than 0.6 (ablation) | AUSAM on ResNet-18 · CIFAR-100 | accuracy: 80.69 -> 80.67 | No clear effect |
| Set sampling rate α to 0.7 rather than 0.6 (ablation) | AUSAM on ResNet-18 · CIFAR-100 | accuracy: 80.69 -> 80.92 | Helped |
Surprising findings
- AUSAM with some settings outperforms the base SGD optimizer and in some cases SAM itself, which the authors attribute to effectively reducing batch size.
- Applying InfoBatch pruning to SAM does not preserve SAM's accuracy on CIFAR-100, while random halving (SAM+Random) performs better than InfoBatch variants.
Limitations
- Speedup is limited by non-forward/backward overheads such as data loading, so AUSAM cannot reach SGD speed at small alpha.
- Benefits are small for small networks like ResNet-18, where forward/backward time is a smaller share of training time.
- Some baseline numbers are taken from original papers rather than re-run under identical settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet MPII
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet MPII
Metrics: accuracy SAM% training speed ratio PCKh
Baselines: SGD SAM LookSAM ESAM SAF MESA K-SAM G-RST InfoBatch Adam
Builds on: Sharpness-Aware Minimization InfoBatch ESAM LookSAM
Closest papers by meaning
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.90)
- Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.87)
- Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification (ICML 2025 · similarity 0.86)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.86)
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.86)
- Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.