Research map / Label-efficient and robust learning / Knowledge distillation
Distilling the Knowledge in Data Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that distilling from a teacher trained on the full dataset (self-distillation) into a student trained on a pruned subset improves accuracy across pruning methods and fractions, making random pruning competitive with score-based pruning.
Problem
Data pruning methods struggle to match full-data accuracy, especially at high pruning rates (low retained fraction f), and score-based methods retain noisy or ambiguous samples.
Approach
Use a teacher pre-trained on the full dataset to provide soft predictions via knowledge distillation while training a student on the pruned subset, with the KD weight adapted to the pruning factor; random pruning plus KD becomes competitive with sophisticated pruning methods.
Key results
- On CIFAR-100 with f=0.1, KD raises random-pruning accuracy from 39.8% to 56.8% (+17%).
- On ImageNet with f=0.1, Top-5 accuracy rises from 82.37% to 87.19% with random pruning under KD.
- With KD, random pruning on ImageNet using only 50% of the data achieves superior accuracy to the full-data baseline; comparable accuracy with 30% (+1.6% at 70%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add knowledge distillation to random-pruned training at pruning factor f=0.1 | ResNet-32 student · CIFAR-100 | accuracy: 39.8% to 56.8% | Helped |
| Add knowledge distillation to random-pruned training at pruning factor f=0.1 | ResNet-50 student · ImageNet | Top-5 accuracy: 82.37% to 87.19% | Helped |
| Add knowledge distillation to EL2N-pruned training at pruning factor f=0.1 | ResNet-50 student · ImageNet | Top-5 accuracy: 62.47% to 82.47% | Helped |
| Increase KD weight α from 0.5 to 1 (ablation) | Forgetting-pruned student · CIFAR-100, f=0.1 | accuracy: more than 8% higher accuracy | Helped |
| Decrease KD weight α from 1 to 0.5 (ablation) | Forgetting-pruned student · CIFAR-100, f=0.7 | accuracy: more than 1 % accuracy | Helped |
| Use a higher-capacity teacher (ablation) | Random-pruned student · CIFAR-100, low pruning fractions | accuracy | Hurt |
| Add VID loss to vanilla KD (ablation) | Forgetting-pruned ResNet-32 student · CIFAR-100, 5% retained | accuracy: 30.5 -> 32.5 | Helped |
| Add VID loss to vanilla KD (ablation) | Forgetting-pruned ResNet-32 student · CIFAR-100, 10% retained | accuracy: 47.68 -> 49.46 | Helped |
| Add VID loss to vanilla KD (ablation) | Forgetting-pruned ResNet-32 student · CIFAR-100, 30% retained | accuracy: 66.15 -> 67.38 | Helped |
| Add VID loss to vanilla KD (ablation) | Forgetting-pruned ResNet-32 student · CIFAR-100, 50% retained | accuracy: 70.95 -> 71.16 | Helped |
Surprising findings
- At low pruning fractions, larger teachers degrade student accuracy; smaller teachers (even smaller than the student) yield better results.
- With KD, simple random pruning matches or beats sophisticated score-based pruning methods across pruning regimes.
Limitations
- Theory is limited to regularized linear regression, with extension to logistic regression only empirical.
- Experiments are on image classification with small ResNet-style models on CIFAR/SVHN and ImageNet; no other domains tested.
- Optimal KD weight alpha must be tuned per pruning fraction.
Experimental setup
Datasets: CIFAR-10 SVHN CIFAR-100 ImageNet
Benchmarks: CIFAR-10 SVHN CIFAR-100 ImageNet
Metrics: top-1 accuracy top-5 accuracy
Baselines: forgetting GraNd EL2N memorization random pruning FitNets AB AT FT FSP NST PKT
Builds on: knowledge distillation forgetting EL2N GraNd DeepCore RepDistiller
Closest papers by meaning
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.91)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.90)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.89)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.