Research map / Label-efficient and robust learning / Knowledge distillation
Prune Your Model before Distill It
Jinhyuk Park, Albert No
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune the VGG19 teacher (36% sparsity) before distilling (ablation) | vanilla KD, VGG19 to VGG11 · CIFAR100 | accuracy: 72.02 -> 72.76 | Improved |
| prune the ResNet18 teacher (36% sparsity) before self distillation (ablation) | vanilla KD, ResNet18 self distillation · TinyImageNet | accuracy: 57.97 -> 59.39 | Improved |
| prune the ResNet18DBL teacher (59% sparsity) before self distillation (ablation) | vanilla KD, ResNet18DBL · TinyImageNet | accuracy: 61.35 -> 63.19 | Improved |
| prune the ResNet18 teacher (79% sparsity) when the student is MobileNetV2 (ablation) | vanilla KD, ResNet18 to MobileNetV2 · TinyImageNet | accuracy: 56.10 -> 57.20 | Improved |
| prune the ResNet18 teacher (36% sparsity) when the student is VGG16 (ablation) | vanilla KD, ResNet18 to VGG16 · TinyImageNet | accuracy: 54.75 -> 56.35 | Improved |
| use a student whose layer widths match the 59% pruned teacher instead of hand-designed halved students (ablation) | prune-then-distill, VGG19 · CIFAR100 | accuracy: 71.43 -> 73.81 | Improved |
| prune the teacher to 79% for ResNet18-ST79 student (vs unpruned teacher) (ablation) | prune-then-distill, ResNet18-ST79 · TinyImageNet | accuracy: 55.65 -> 56.23 | Improved |
| prune the ResNet50 teacher (59% sparsity) in self distillation (ablation) | vanilla KD, ResNet50 · TinyImageNet | accuracy: 64.54 -> 65.21 | Improved |
| prune the ResNet50 teacher to 59% for the constructed student ResNet50-ST59 (ablation) | prune-then-distill, ResNet50-ST59 · TinyImageNet | accuracy: 63.84 -> 63.58 | Hurt |
| prune the ResNet50 teacher to 36% for ResNet50-ST36 (ablation) | prune-then-distill, ResNet50-ST36 · TinyImageNet | accuracy: 64.11 -> 64.12 | No clear effect |
| prune the ResNet18 teacher to 36% for ResNet18-ST36 on ImageNet (ablation) | prune-then-distill, ResNet18-ST36 · ImageNet | accuracy: 60.93 -> 61.10 | Improved |
| prune the ResNet18 teacher to 79% for ResNet18-ST79 on ImageNet (ablation) | prune-then-distill, ResNet18-ST79 · ImageNet | accuracy: 50.24 -> 50.14 | No clear effect |
Closest papers by meaning
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.91)
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.90)
- Few Sample Knowledge Distillation for Efficient Network Compression (CVPR 2020 · similarity 0.90)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.89)
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (NeurIPS 2020 · similarity 0.89)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
