Research map / Architectures and efficiency / Pruning and efficient tuning
Comparing Rewinding and Fine-tuning in Neural Network Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fine-tuning with weight rewinding | Pruned networks across the studied architectures and datasets · CIFAR-10, ImageNet, and WMT16 EN-DE | accuracy | Helped |
| Replace weight rewinding with learning rate rewinding | Pruned networks across the studied architectures and datasets · CIFAR-10, ImageNet, and WMT16 EN-DE | accuracy | Helped |
| Use iterative unstructured pruning with learning rate rewinding for the original training time | ResNet-50 · ImageNet, no accuracy drop | compression ratio: 5.96× | Helped |
| Replace fine-tuning with either rewinding technique | Pruned networks · Unstructured and structured pruning across studied networks | accuracy at equivalent retraining epochs | Helped |
| Rewind weights all the way to initialization using weight rewinding | Unstructured-pruned networks · Unstructured pruning, full original training time | accuracy | Hurt |
| Use learning rate rewinding rather than fine-tuning | Iteratively pruned vision networks · ImageNet, ResNet-50 | theoretical speedup: 1.7× -> 4.8× | Helped |
| Rewind weights but use the fine-tuning learning rate (ablation) | Pruned networks (low-LR weight rewinding) · Unstructured pruning | accuracy | Hurt |
| Reinitialize the pruned network and retrain it (ablation) | Networks with discovered pruned structure · Unstructured pruning | accuracy | Hurt |
| Reinitialize the pruned network and retrain it (ablation) | Networks with discovered pruned structure · Structured pruning | accuracy | No clear effect |
| Extend reinitialization training with extra retraining epochs (ablation) | Reinitialized pruned networks · Structured and unstructured pruning | accuracy | No clear effect |
| Retrain longer with low-LR weight rewinding (ablation) | Pruned networks · Different networks and compression ratios | accuracy | Mixed |
| Retrain with learning rate rewinding for the full original training time | Pruned networks · One-shot pruning across studied networks | accuracy | Helped |
Closest papers by meaning
- Network Pruning That Matters: A Case Study on Retraining Variants (ICLR 2021 · similarity 0.93)
- How I Learned to Stop Worrying and Love Retraining (ICLR 2023 · similarity 0.87)
- “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018 · similarity 0.87)
- Pruning Neural Networks at Initialization: Why Are We Missing the Mark? (ICLR 2021 · similarity 0.86)
- Pruning from Scratch (AAAI 2020 · similarity 0.86)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.