Research map / Architectures and efficiency / Pruning and efficient tuning
A Three-regime Model of Network Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase dense-training temperature (for example, use fewer training epochs) at low model density | PreResNet-20 with UniformMP · CIFAR-10, low density | test error | Helped |
| Decrease dense-training temperature at high model density | PreResNet-20 with UniformMP · CIFAR-10, high density | test error | Helped |
| Use LMC to select a dense model for pruning rather than relying only on its dense-model test error | PreResNet-20 model selection · CIFAR-10, low-density pruning | test error | Helped |
| Use LMC-based selection instead of dense-model test-error selection | PreResNet-20 model selection · CIFAR-10, high-density pruning | test error | No clear effect |
| Use SAM with a large neighborhood size ρ | PreResNet-20 with UniformMP · CIFAR-10, low densities 5%, 7%, and 10% | test error | Helped |
| Use SAM with a large neighborhood size ρ | PreResNet-20 with UniformMP · CIFAR-10, high densities 20%, 40%, and 80% | test error | Hurt |
| Select the SAM neighborhood size ρ using CKA similarity | PreResNet-20 with SAM · CIFAR-10, varying model densities | test error | Helped |
| Use higher temperature (fewer training epochs) for low-density pruning and lower temperature for high-density pruning | Transformer-base with UniformMP · WMT14 DE-EN validation | BLEU | Mixed |
| Use larger temperature by reducing dense-training epochs (ablation) | PreResNet-20 with Adam and/or SGD during pruning · CIFAR-10, densities 5% and 6% | test error | Helped |
| Use Adam rather than SGD for retraining pruned models (ablation) | PreResNet-20 · CIFAR-10, densities 5% and 6% | test error | Helped |
| Use Adam rather than SGD for dense-model training (ablation) | PreResNet-20 · CIFAR-10, densities 5% and 6% | test error | Hurt |
| Replace UniformMP with GlobalMP (ablation) | PreResNet-20 · CIFAR-10, varying density and training epochs | test error | No clear effect |
Closest papers by meaning
- A Gradient Flow Framework For Analyzing Network Pruning (ICLR 2021 · similarity 0.91)
- The Generalization-Stability Tradeoff In Neural Network Pruning (NeurIPS 2020 · similarity 0.90)
- On the Predictability of Pruning Across Scales (ICML 2021 · similarity 0.90)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.