Research map / Architectures and efficiency / Pruning and efficient tuning
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace magnitude pruning with second-order WoodFisher pruning | ResNet-50 gradual pruning to 95% sparsity · ImageNet val | Top-1 accuracy: by 0.4% Top-1 validation accuracy | Improved |
| add SGD fine-tuning between pruning steps | WoodFisher vs magnitude pruning · ImageNet val (ResNet-50) | accuracy gap between methods | No clear effect |
| replace SNIP with GraSP (gradient-flow-aware pruning) | pruning at initialization · very sparse regimes | accuracy | Improved |
| use SNIP at sparsity above 99% | pruning at initialization · high sparsity | accuracy | Hurt |
| use iterative SNIP-it variant with more data in structure search | pruning before training · comparison with GraSP | performance cost: 5x improvement of performance over GraSP | Improved |
Closest papers by meaning
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.94)
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.93)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.91)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.90)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
