AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

A statistical perspective on distillation

Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, Sanjiv Kumar

ICML 2021 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 8 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
standard distillation from teacher logitsstudent feedforward model · AmazonCat-13K and AmazonCat-670KPrecision@k: small but consistent bump over student baselineImproved
negative-aware distillation (smoothing negative labels with teacher probabilities)student trained with teacher logits · AmazonCat-13KP@1, P@3, P@5: P@1 0.8131 to 0.8560; P@3 0.6363 to 0.7148; P@5 0.4918 to 0.5715Improved
negative-aware distillationstudent trained with teacher logits · AmazonCat-670KP@1, P@3, P@5: P@1 +0.0019 over SD; P@3 +0.0010; P@5 -0.0027Mixed
Bayes teacher (distilling with true class probabilities) vs one-hot labels (ablation)linear logistic regression student on synthetic Gaussian data · synthetic binary Gaussian (Figure 2)test accuracy: significant gain for small NImproved
teacher probability quality (lower MSE vs Bayes) via alpha distortion, holding teacher accuracy fixed (ablation)linear student on synthetic data · synthetic (Figure 3)student accuracy: student accuracy best at alpha=1.0 and tracks teacher MSEImproved
temperature scaling of teacher (tuning T to improve teacher probability quality) (ablation)ResNet-14 student distilled from ResNet-32 teacher · CIFAR-100 (Figure 4)student accuracy: optimal teacher MSE at T=2; optimal student accuracy at T=3Improved
choosing teacher with better probability estimates (MSE) over more accurate teacher (deeper ResNet) (ablation)ResNet-8 student from ResNet teachers of varying depth · CIFAR-100 (Figure 1)teacher accuracy vs student accuracy: teacher accuracy rises with depth while probability quality degrades beyond depth 14; students track probability qualityMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy