Research map / Architectures and efficiency / Pruning and efficient tuning
Adaptive Sharpness-Aware Pruning for Robust Sparse Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaSAP, a three-step pruning procedure that uses adaptive (importance-scaled) sharpness-aware weight perturbations during warmup, prunes channels, then fine-tunes with uniform ASAM perturbations to produce sparse networks robust to unseen corruptions and distribution shifts.
Problem
Structured pruning of deep networks degrades out-of-distribution robustness (ImageNet-C, ImageNet-V2, corrupted Pascal VOC) more than validation accuracy, and robustness to unseen input variation is not preserved by existing pruning methods.
Approach
Scale per-neuron SAM-style perturbation radius inversely with neuron importance during warmup so unimportant neurons are pushed into flat regions before pruning, then apply uniform flatness-based (ASAM) optimization after pruning to encourage robustness; the method is pruning-criterion agnostic.
Key results
- AdaSAP_P on ResNet50 at 0.20 size reaches 37.30% on ImageNet-C vs 33.93% for Taylor pruning (about +3.4 points).
- At 0.40 size on ResNet50, AdaSAP_P gets 41.23% IN-C and 77.27% val, versus 37.84% IN-C for Taylor at 0.42 size.
- On Pascal VOC-C, AdaSAP improves robust mAP by about 4% over HALP according to the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply AdaSAP_P with magnitude pruning rather than magnitude pruning alone | ResNet50 · ImageNet-C, size 0.20 | Top1 accuracy: 35.27 -> 37.30 | Helped |
| apply AdaSAP_P with magnitude pruning rather than Taylor pruning | ResNet50 · ImageNet-C, size 0.20 | Top1 accuracy: 33.93 -> 37.30 | Helped |
| apply AdaSAP_L rather than HALP | ResNet50 · ImageNet-C, speedup 2.6 | Top1 accuracy: 35.03 -> 37.93 | Helped |
| apply AdaSAP_L rather than HALP | ResNet50 · ImageNet-C, speedup 1.6 | Top1 accuracy: 39.13 -> 41.63 | Helped |
| apply AdaSAP_L rather than SMCP | ResNet50 · ImageNet-C, speedup 1.2 | Top1 accuracy: 40.91 -> 42.92 | Helped |
| add adaptive warmup perturbations instead of using Taylor pruning with SAM (ablation) | ResNet50 · ImageNet-C, size 0.20 | Top1 accuracy: 34.49 -> 35.86 | Helped |
| use uniform rather than adaptive perturbations during robustness encouragement (ablation) | ResNet50 · ImageNet-C | Top1 accuracy: 35.72 -> 35.86 | Helped |
| remove weight perturbations during robustness encouragement (ablation) | AdaSAP · ImageNet-C | Top1 accuracy: 35.84 | No clear effect |
| extend adaptive weight perturbation training from 5 to 30 epochs (ablation) | AdaSAP_P · ImageNet-C | Top1 accuracy: 35.65 -> 35.04 | Hurt |
| use adaptive weight perturbations rather than Taylor pruning with SAM (ablation) | ResNet50 · ImageNet-V2, size 0.20 | Top1 accuracy: 61.37 -> 62.03 | Helped |
Surprising findings
- Adaptive weight perturbations alone during robustness encouragement did not beat uniform perturbations in the ablation (IN-C 35.72 vs 35.86).
- Pruning ratio increases degrade robustness more than validation accuracy for SOTA pruning methods, even as dense models retain about 55% relative robustness.
Limitations
- Roughly doubles training cost due to two backward passes per step.
- Robustness evaluated only on existing synthetic corruption and distribution-shift datasets, not real-world autonomous driving corruptions.
- Focused on structured channel pruning; unstructured pruning and from-scratch pruning not evaluated.
Experimental setup
Datasets: ImageNet-1K Pascal VOC
Benchmarks: ImageNet-1K validation ImageNet-C ImageNet-V2 Pascal VOC Pascal VOC-C
Metrics: Top-1 accuracy mAP robustness ratio (R_C, R_V2)
Baselines: Taylor pruning HALP Magnitude pruning EagleEye ABCPruner SMCP GReg SAM
Builds on: Sharpness-Aware Minimization ASAM HALP Taylor pruning magnitude pruning
Closest papers by meaning
- SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · similarity 0.90)
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.89)
- CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models (NeurIPS 2023 · similarity 0.89)
- Holistic Adversarially Robust Pruning (ICLR 2023 · similarity 0.88)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.87)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.