Research map / Architectures and efficiency / Pruning and efficient tuning
EPSD: Early Pruning with Self-Distillation for Efficient Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EPSD, which prunes a randomly initialized network at early stage using saliency computed from the self-distillation loss, then trains the sparse network with self-distillation, achieving competitive accuracy with two-step compression.
Problem
Combining early pruning with self-distillation naively ("Simple Combination") causes severe accuracy degradation at high sparsity, and conventional prune-then-distill pipelines need costly teacher pre-training and multiple steps.
Approach
Rank weights at initialization by gradient-based saliency computed on the self-distillation loss (after a few update steps) rather than the cross-entropy loss, so the pruned sub-network retains "distillable" weights; then train it with SD in a two-step pipeline with no pre-training.
Key results
- On ImageNet with ResNet-50 at 90% sparsity, EPSD reaches 66.3% top-1 (DLB variant), versus 66.1% for ProsPr-style early pruning baseline in the table (ProsPr 65.9%) and 65.9%-ish range for other early pruning methods; Simple Combination collapses to 8.3-9.9%.
- On CIFAR-100 with ResNet-18 at 95% sparsity, Simple Combination drops to 62.67% accuracy, while EPSD stays competitive.
- On CUB-200-2011 weakly supervised localization at 50% sparsity, EPSD achieves 24.40% classification error and 41.23% top-1 localization error versus ProsPr's 25.39% and 48.65%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SD-loss gradients to select and preserve distillable weights during early pruning instead of the simple pruning-plus-SD sequence | early-pruned ResNet-18 · CIFAR-100, 95% sparsity | accuracy: 62.67% | Helped |
| Use EPSD rather than Simple Combination for early pruning with DLB | VGG-19 · Tiny-ImageNet, 90% sparsity | accuracy: 29.08% vs. 49.88%; 53.91% | Helped |
| Use EPSD rather than ProsPr for unstructured pruning | ResNet-50 · ImageNet, 90% sparsity | top-1 accuracy: 65.9 -> 66.3 | Helped |
| Use EPSD rather than the unpruned baseline | VGG-19 · CIFAR-100, 36% sparsity | accuracy: 58.64% vs. 56.45% | Helped |
| Add SD-based pre-training before pruning with CE and SD fine-tuning (Simple Combination-2) (ablation) | ResNet-50 compression · ImageNet, 90% sparsity | top-1 accuracy: 66.2% vs. 66.4% | Helped |
| Use SD-loss saliency instead of CE-loss saliency when pruning an SD-pretrained model (ablation) | ResNet-50 compression · ImageNet, 90% sparsity | top-1 accuracy: 66.4% -> 66.6% | Helped |
| Use SD-loss pruning rather than CE-loss pruning before SD fine-tuning (ablation) | ResNet-50 · ImageNet, 90% sparsity | top-1 accuracy: 9.9 -> 66.2 | Helped |
| Add EPSD's SD-based pruning to RigL | WideResNet-22-2 · CIFAR-10, 90% sparsity | accuracy: 92.16% -> 92.49% | Helped |
| Replace one-shot EPSD with iterative EPSD (ablation) | ResNet-18 · CIFAR-10, 80% sparsity | expected calibration error and AURC | Helped |
| Use standard cross-entropy saliency rather than SD-loss saliency during pruning (ablation) | SD-pretrained ResNet-50 · ImageNet, 90% sparsity | top-1 accuracy: 66.6% vs. 66.4% | Hurt |
| Use EPSD instead of Simple Combination for structured pruning | VGG-19 · CIFAR-100, 80% sparsity | accuracy: 72.29 -> 73.16 | Helped |
Surprising findings
- The naive combination of early pruning and self-distillation performs far worse than either component alone at high sparsity (e.g., 9.9% top-1 on ImageNet ResNet-50 at 90% sparsity).
- An 80%-sparse EPSD model outperforms several unpruned SD methods (e.g., ResNet-32 on CIFAR-10: 94.68% vs 93.46% unpruned baseline).
Limitations
- Evaluation focuses on vision models; large-scale language or multimodal networks are not studied.
- Only one early pruning method (ProsPr) and SNIP are used as the pruning backbone; the approach is tied to gradient saliency.
- Comparison against SD methods uses 80% sparsity while the SD baselines are unpruned, making the comparison asymmetric.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet CUB-200-2011 Pascal VOC 2012
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet CUB-200-2011 Pascal VOC 2012
Metrics: top-1 accuracy top-5 accuracy mean IoU pixel accuracy classification error localization error Mean-JSV ECE AURC
Baselines: SNIP GraSP FORCE DOP ProsPr 3SP Random structured pruning LSR TFKD CSKD PSKD DLB
Builds on: SNIP ProsPr Prune then Distill PS-KD CS-KD DLB
Closest papers by meaning
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.91)
- Few Sample Knowledge Distillation for Efficient Network Compression (CVPR 2020 · similarity 0.90)
- “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018 · similarity 0.90)
- Network Pruning via Performance Maximization (CVPR 2021 · similarity 0.89)
- Computation and Memory-Efficient Model Compression with Gradient Reweighting (NeurIPS 2025 · similarity 0.89)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.