Research map / Architectures and efficiency / Pruning and efficient tuning
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
Aleksandra Nowak, Bram Grooten, Decebal Constantin Mocanu, Jacek Tabor
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SNIP (Taylor/gradient) instead of magnitude pruning | DST · CIFAR10/CIFAR100/Tiny-ImageNet models at low density | test accuracy | Hurt |
| use SNIP instead of magnitude pruning at density 0.9 (random growth) | DST on RoBERTa Large · CommonsenseQA | accuracy: 73.80 -> 56.57 | Hurt |
| use RSensitivity instead of magnitude pruning at high density | DST · CIFAR10 large-MLP and ResNet models, density > 0.15 | test accuracy | Improved |
| use RSensitivity instead of magnitude at low density (random growth, density 0.9) | DST on RoBERTa Large · CommonsenseQA | accuracy: 73.80 -> 25.87 | Hurt |
| use MEST instead of magnitude pruning | DST · vision models | test accuracy | No clear effect |
| increase MEST lambda toward infinity (ablation) | MEST · vision models | test accuracy | Hurt |
| use SET instead of magnitude pruning | DST · CIFAR10 first-update masks | mask overlap | No clear effect |
| DST instead of static sparse training at low density | sparse training · vision/tabular models | test accuracy | Improved |
| increase update period from small to 800 (random growth) (ablation) | DST · MLP CIFAR10 density 0.2/0.1 | validation accuracy | Improved |
| increase update period to 1600 (gradient growth) (ablation) | DST · MLP CIFAR10 | validation accuracy | Improved |
| use very small update period (25) with gradient-based pruning and gradient growth (ablation) | DST · CIFAR10 | validation accuracy | Hurt |
| choose gradient growth over random growth | DST · convolutional models | test accuracy | No clear effect |
Closest papers by meaning
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.92)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.89)
- Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · similarity 0.89)
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.88)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.87)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
