Research map / Label-efficient and robust learning / Knowledge distillation
A statistical perspective on distillation
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, Sanjiv Kumar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| standard distillation from teacher logits | student feedforward model · AmazonCat-13K and AmazonCat-670K | Precision@k: small but consistent bump over student baseline | Improved |
| negative-aware distillation (smoothing negative labels with teacher probabilities) | student trained with teacher logits · AmazonCat-13K | P@1, P@3, P@5: P@1 0.8131 to 0.8560; P@3 0.6363 to 0.7148; P@5 0.4918 to 0.5715 | Improved |
| negative-aware distillation | student trained with teacher logits · AmazonCat-670K | P@1, P@3, P@5: P@1 +0.0019 over SD; P@3 +0.0010; P@5 -0.0027 | Mixed |
| Bayes teacher (distilling with true class probabilities) vs one-hot labels (ablation) | linear logistic regression student on synthetic Gaussian data · synthetic binary Gaussian (Figure 2) | test accuracy: significant gain for small N | Improved |
| teacher probability quality (lower MSE vs Bayes) via alpha distortion, holding teacher accuracy fixed (ablation) | linear student on synthetic data · synthetic (Figure 3) | student accuracy: student accuracy best at alpha=1.0 and tracks teacher MSE | Improved |
| temperature scaling of teacher (tuning T to improve teacher probability quality) (ablation) | ResNet-14 student distilled from ResNet-32 teacher · CIFAR-100 (Figure 4) | student accuracy: optimal teacher MSE at T=2; optimal student accuracy at T=3 | Improved |
| choosing teacher with better probability estimates (MSE) over more accurate teacher (deeper ResNet) (ablation) | ResNet-8 student from ResNet teachers of varying depth · CIFAR-100 (Figure 1) | teacher accuracy vs student accuracy: teacher accuracy rises with depth while probability quality degrades beyond depth 14; students track probability quality | Mixed |
Closest papers by meaning
- On student-teacher deviations in distillation: does it pay to disobey? (NeurIPS 2023 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification (ICML 2024 · similarity 0.88)
- Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · similarity 0.88)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
