Research map / Label-efficient and robust learning / Knowledge distillation
What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective
Huan Wang, Suhas Lohit, Michael Jeffrey Jones, Yun Fu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CutMix augmentation to KD | KD · CIFAR100 (resnet32x4 → ShuffleV2) | accuracy: 74.45 -> 76.61 | Improved |
| Add teacher-entropy CutMixPick to KD+CutMix (ablation) | KD · CIFAR100 (resnet32x4 → ShuffleV2) | accuracy: 76.61 -> 76.90 | Improved |
| Train KD+CutMixPick for 960 rather than 240 epochs (ablation) | KD+CutMixPick · CIFAR100 (resnet32x4 → ShuffleV2) | accuracy: 76.90 -> 77.90 | Improved |
| Add CutMixPick to CRD | CRD · CIFAR100 (resnet32x4 → ShuffleV2) | accuracy: 76.05 -> 76.93 | Improved |
| Add CutMixPick to CRD | CRD · Tiny ImageNet (resnet32x4 → ShuffleV2) | accuracy: 67.33 -> 67.64 | Improved |
| Add teacher-entropy CutMixPick to KD+CutMix (ablation) | KD · Tiny ImageNet (resnet56 → resnet20) | accuracy: 53.77 -> 53.66 | Hurt |
| Select augmented CutMix samples using teacher entropy rather than student entropy (ablation) | KD+CutMixPick · CIFAR100 | test loss: 1.0270 -> 0.9570 | Improved |
| Select augmented CutMix samples using student entropy rather than teacher entropy (ablation) | KD+CutMixPick · ImageNet | test loss: 1.1955 -> 1.2049 | Hurt |
| Use lower T. stddev augmentation schemes | KD · CIFAR100 and Tiny ImageNet | student test loss | Improved |
| Use T. stddev to rank augmentation quality | KD · ImageNet | student test loss | No clear effect |
| Add CutMix to other KD methods | AT, CC, SP, PKT, and VID · CIFAR100 | accuracy | Improved |
Closest papers by meaning
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.88)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.88)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
