Research map / Label-efficient and robust learning / Knowledge distillation
Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning
Yifan Wu, Yiqi Wang, Xichen Ye, Wenjing Yan, Xiaoqiang Li, Cheng Jin, WEIZHONG ZHANG, Xiangyu Yue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed-threshold CCS selection with learnable Beta policy (ablation) | IF-CCS · CIFAR-100 KD (prune 50%) | accuracy: 77.64 -> 79.10 | Improved |
| replace fixed-window BWS selection with learnable Beta policy (ablation) | IF-BWS · CIFAR-10 KD (prune 90%) | accuracy: 86.20 -> 88.51 | Improved |
| replace fixed-window BWS selection with learnable Beta policy (ablation) | IF-BWS · CIFAR-100 KD (prune 90%) | accuracy: 58.71 -> 64.11 | Improved |
| replace fixed-window BWS selection with learnable Beta policy (ablation) | IF-BWS · ImageNet KD (prune 50%) | accuracy: 73.63 -> 74.02 | Improved |
| prune 30% of CIFAR-10 with IF-Beta instead of full data | IF-Beta · CIFAR-10 KD | accuracy: 95.50 -> 95.69 | Improved |
| prune data with IF-Beta instead of full data | IF-Beta · CIFAR-100 KD | accuracy: 79.38 -> 79.53 | Improved |
| prune 50% of ImageNet with IF-Beta instead of full data | IF-Beta · ImageNet KD | accuracy: 73.54 -> 74.02 | Improved |
| select data with IF-Beta instead of random pruning under limited budget | IF-Beta · CIFAR-100 KD limited budget (prune 50%) | accuracy: 63.39 -> 67.31 | Improved |
| use generic ImageNet-pretrained model instead of in-domain model for influence scores (no KD) (ablation) | IF-Beta · CIFAR-100 standard pruning (prune 90%) | accuracy: 55.78 -> 53.31 | Hurt |
| use IF-Beta instead of Medium-Difficulty* for efficient KD | IF-Beta · ImageNet KD ResNet-34 to MobileNet | accuracy: 70.92 -> 71.20 | Improved |
| use IF-Beta instead of full-data baseline for training time | IF-Beta · ImageNet KD ResNet-34 to MobileNet | training time (h): 39.86 -> 19.24 | Improved |
| use IF-Beta instead of random pruning with FitNets loss | IF-Beta · CIFAR-100 FitNets KD (prune 90%) | accuracy: 64.99 -> 71.52 | Improved |
Closest papers by meaning
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.92)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.90)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
