Research map / Architectures and efficiency / Pruning and efficient tuning
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune 5% of weights in Q, K, V, O, and FFN | ESM-1B · HotProtein HP-S²C2 | accuracy: 94.68 -> 92.19 | Hurt |
| Prune 5% of weights in Q, K, and V (ablation) | ESM-1B · HotProtein HP-S²C2 | accuracy: 94.68 -> 92.55 | Hurt |
| Prune 5% of weights in Q and K (ablation) | ESM-1B · HotProtein HP-S²C2 | accuracy: 94.68 -> 93.62 | Hurt |
| Prune 5% of weights in Q and V (ablation) | ESM-1B · HotProtein HP-S²C2 | accuracy: 94.68 -> 93.62 | Hurt |
| Prune 5% of weights in K and V (ablation) | ESM-1B · HotProtein HP-S²C2 | accuracy: 94.68 -> 92.55 | Hurt |
| Use second-order oBERT pruning with learning-rate rewinding and knowledge distillation instead of magnitude-based pruning | RoBERTa-Large · CSQA | accuracy | Helped |
| Prune at higher sparsity on the harder CSQA task rather than the easier WinoGrande and RACE tasks | Sparse RoBERTa · CSQA, WinoGrande, and RACE | accuracy: up to 74%; 3% ~ 10% | Hurt |
| Use post-training pruning rather than prior-training pruning | Sparse models on commonsense reasoning · CSQA, WinoGrande, and RACE | accuracy | Helped |
| Add RigL topology updates to prior-training pruning (ablation) | Prior-training sparse models · CSQA, WinoGrande, and RACE | accuracy | No clear effect |
| Use iterative magnitude pruning rather than one-shot or gradual magnitude pruning at high sparsity | GTS and Graph2Tree · MAWPS, ASDiv-A, and SVAMP | accuracy | Hurt |
| Prune before training using magnitude (OMP Before) | GTS and Graph2Tree · MAWPS, ASDiv-A, and SVAMP | accuracy: 80% to nearly 0% | Hurt |
| Increase sparsity beyond 20% | ESM-1B · HotProtein HP-S and Meltome Atlas | accuracy | Hurt |
Closest papers by meaning
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.92)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.90)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.90)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.90)
- Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.