Research map / Architectures and efficiency / Pruning and efficient tuning
SAFE: Finding Sparse and Flat Minima to Improve Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAFE, an ADMM-based sparsity-constrained optimization method that jointly enforces weight sparsity and flat minima during training (and SAFE+ with a generalized saliency-based projection), evaluated on CIFAR image classification and LLaMA post-training pruning.
Problem
Sparsifying neural networks typically degrades performance, and restoring accuracy after pruning is difficult; the paper asks whether explicitly seeking sparse and flat minima can recover it.
Approach
Formulates pruning as a min-max sparsity-constrained problem (SAM-style flatness inside an L0 constraint) and solves it with an augmented-Lagrangian/ADMM split, where the hard-thresholding projection is generalized with a diagonal metric P so that saliency scores (OBD, SNIP, Wanda) can be plugged in.
Key results
- On CIFAR-100 ResNet-32 at 90% sparsity, SAFE reaches 73.89% vs 70.85% for ADMM and 71.69% for GMP.
- On ResNet-20/CIFAR-10 noisy labels at 90% sparsity with 25% noise, SAFE gets 90.68% vs 79.40% for ADMM.
- On LLaMA-2-7B at 50% sparsity, SAFE+ reaches 6.56 Wikitext perplexity vs 6.99 for SparseGPT and 6.92 for Wanda; SAFE is competitive with ALPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAFE instead of the strongest listed baseline at 99.5% sparsity | CIFAR image classification models · CIFAR-10/100, 99.5% sparsity | validation accuracy: 93.56[0.02] vs 90.63[0.14]; 79.55[0.13] vs 75.83[0.47]; 67.23[0.19] vs 62.77[0.49]; 51.45[0.32] vs 42.93[0.37] | Helped |
| Use SAFE-P with the generalized saliency projection instead of ALPS | LLaMA-2 7B, 50% sparsity · WikiText2 and C4 | perplexity: 6.56[0.01] vs 6.87[0.01]; 8.71[0.00] vs 8.98[0.00] | Helped |
| Use SAFE-P with the generalized saliency projection instead of ALPS | LLaMA-2 13B, 50% sparsity · WikiText2 and C4 | perplexity: 5.67[0.01] vs 5.96[0.02]; 7.74[0.01] vs 8.09[0.04] | Helped |
| Use SAFE-P with the generalized saliency projection instead of ALPS | LLaMA-3 8B, 50% sparsity · WikiText2 and C4 | perplexity: 8.62[0.06] vs 9.05[0.12]; 13.26[0.06] vs 13.40[0.06] | Helped |
| Add flatness enforcement through SAFE instead of ADMM | ResNet-20 · CIFAR-10, 70% sparsity, 25% label noise | validation accuracy: 77.00$_{\pm0.91}$ -> 90.58$_{\pm0.30}$ | Helped |
| Add flatness enforcement through SAFE instead of ADMM | ResNet-20 · CIFAR-10, 70% sparsity, 50% label noise | validation accuracy: 59.18$_{\pm0.55}$ -> 86.51 $_{\pm0.16}$ | Helped |
| Add flatness enforcement through SAFE instead of ADMM | ResNet-20 · CIFAR-10, 70% sparsity, 75% label noise | validation accuracy: 32.62$_{\pm0.89}$ -> 67.01$_{\pm0.54}$ | Helped |
| Use SAFE instead of ADMM for corrupted-image evaluation | Sparse CIFAR-10 models · 90% sparsity; CIFAR-10C and PGD corruptions | accuracy: 70.06 -> 73.98; 52.01 -> 55.11; 49.81 -> 56.43; 49.71 -> 56.36 | Helped |
| Add the projected-point gradient during x-minimization (ablation) | SAFE · ResNet-20, CIFAR-10, 99.5% sparsity | validation accuracy: 79.55$_{\pm 0.13}$ -> 85.85$_{\pm 0.09 }$ | Helped |
Surprising findings
- Magnitude pruning collapses on LLaMA at 60% sparsity (perplexity 1864 on LLaMA-2-7B), while SAFE stays near 9-10.
- CrAM alone performs poorly across sparsities, and CrAM+ gains depend on an auxiliary gradient term that the authors argue confounds attribution to robust optimization.
Limitations
- Hyperparameters such as penalty lambda must be tuned per setting for image classification.
- Theoretical contribution is modest; convergence relies on assumptions (smoothness, weak convexity) and a stationarity notion.
- The SAFE vs. ADMM comparison on robustness and sharpness is limited to a simple MLP and ResNet-20 setup.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 C4 MNIST CIFAR-10C
Benchmarks: CIFAR-10 CIFAR-100 WikiText-2 C4 CIFAR-10C
Metrics: validation accuracy perplexity maximum Hessian eigenvalue adversarial accuracy under PGD
Baselines: GMP PBW LTH MLPrune ADMM SparseGPT Wanda ALPS IMP+SAM CrAM
Builds on: sharpness-aware minimization ADMM CrAM Wanda SparseGPT
Closest papers by meaning
- Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (ICLR 2024 · similarity 0.90)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.90)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.89)
- SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.88)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.88)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.