Research map / Architectures and efficiency / RNNs and normalization
A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S2-SAM, a single-step sharpness-aware minimization that approximates the SAM perturbation with the previous step's gradient, giving zero extra cost for sparse training and improving accuracy of many sparse training methods.
Problem
Sparse training of DNNs reaches sub-optimal generalization, attributed to a chaotic, steep loss surface induced by sparsity; standard SAM improves generalization but doubles training compute, conflicting with sparse training's efficiency goal.
Approach
Replace the gradient at the current weights in SAM's perturbation step with the gradient from the previous iteration, so each iteration needs only one forward/backward pass; add a convergence/generalization analysis under PL condition.
Key results
- On CIFAR-10 ResNet-32 at 98% sparsity, S2-SAM averages +0.77% accuracy across sparse methods (+1.12% on CIFAR-100).
- On ImageNet-1K ResNet-50 at 90% sparsity, S2-SAM improves RigL from 72.00 to 72.44 and MEST (EM) from 73.60 to 74.58 top-1.
- Training throughput with S2-SAM matches original sparse training (e.g., RigL 3098 vs 3133 imgs/s) while SAM drops to 1508 imgs/s; ImageNet-C robustness improves by an average of 3.23%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add S2-SAM to sparse training | Static and dynamic sparse methods · CIFAR-10 and CIFAR-100, ResNet-32, 90%, 95%, and 98% pruning | accuracy: 0.20↑ to 2.07↑ | Helped |
| Add S2-SAM to sparse training | Sparse methods including RigL, MEST, and SNIP · ImageNet-1K, ResNet-50, 80% and 90% sparsity | top-1 accuracy: 0.36↑ to 1.18↑ | Helped |
| Add S2-SAM to structured sparse training | CHEX and Chase · ImageNet-1K | accuracy: 0.44↑; 0.51↑; 0.43↑; 0.55↑ | Helped |
| Use single-step S2-SAM instead of two-step SAM | RigL sparse training · ImageNet-1K, 90% sparsity | throughput: 1508 imgs/s -> 3098 imgs/s | Helped |
| Add S2-SAM to sparse training | SNIP, GraSP, MEST (EM), and RigL · ImageNet-C, 80% sparsity | accuracy: 3.75↑; 2.93↑; 3.11↑; 3.12↑ | Helped |
| Apply S2-SAM to dense training | ResNet-32, MobileNet-V2, and VGG-19 · CIFAR-10 | accuracy: 0.41↑; 0.42↑; 0.27↑ | Helped |
| Apply S2-SAM to dense training | EfficientNet-B0, ResNet-34, and ResNet-50 · ImageNet-1K | accuracy: 0.56↑; 0.49↑; 0.42↑ | Helped |
| Add S2-SAM to sparse training | Static and dynamic sparse methods · CIFAR-10 and CIFAR-100, VGG-19, 90%, 95%, and 98% pruning | accuracy: 0.25↑ to 1.99↑ | Helped |
| Add S2-SAM to sparse training | Sparse methods at 98% sparsity · CIFAR-10 and CIFAR-100, ResNet-32 | accuracy: 0.77%; 1.12% | Helped |
| Add S2-SAM to sparse training | SNIP, GraSP, and MEST · CIFAR-10, ResNet-32, loss surface visualization | Ra | Helped |
Surprising findings
- S2-SAM also improves dense model training (e.g., ResNet-50 on ImageNet-1K from 76.90 to 77.32), with smaller networks benefiting more.
- Gains in accuracy on ImageNet-1K at 90% sparsity are smaller than the SAM gain in some cases, and SAM slightly outperforms S2-SAM in accuracy while being about twice as costly.
Limitations
- Focus on unstructured/structured sparse training on image classification; dense-training results are limited and the authors defer broader study.
- Theoretical analysis relies on assumptions (bounded gradients, PL condition, smoothness) and the generalization bound is loose in practice.
- Code is promised only after acceptance; no release at submission time.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K ImageNet-C
Metrics: top-1 test accuracy training FLOPs inference FLOPs training throughput (imgs/s) Ra sharpness coefficient
Baselines: LT SNIP GraSP SET DSR RigL MEST SAM CHEX Chase Top-KAST ESAM
Builds on: Sharpness-Aware Minimization lottery ticket hypothesis RigL MEST
Closest papers by meaning
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.92)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.91)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.91)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.89)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.89)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.