AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective

Huan Wang, Suhas Lohit, Michael Jeffrey Jones, Yun Fu

NeurIPS 2022 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 16 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add CutMix augmentation to KDKD · CIFAR100 (resnet32x4 → ShuffleV2)accuracy: 74.45 -> 76.61Improved
Add teacher-entropy CutMixPick to KD+CutMix (ablation)KD · CIFAR100 (resnet32x4 → ShuffleV2)accuracy: 76.61 -> 76.90Improved
Train KD+CutMixPick for 960 rather than 240 epochs (ablation)KD+CutMixPick · CIFAR100 (resnet32x4 → ShuffleV2)accuracy: 76.90 -> 77.90Improved
Add CutMixPick to CRDCRD · CIFAR100 (resnet32x4 → ShuffleV2)accuracy: 76.05 -> 76.93Improved
Add CutMixPick to CRDCRD · Tiny ImageNet (resnet32x4 → ShuffleV2)accuracy: 67.33 -> 67.64Improved
Add teacher-entropy CutMixPick to KD+CutMix (ablation)KD · Tiny ImageNet (resnet56 → resnet20)accuracy: 53.77 -> 53.66Hurt
Select augmented CutMix samples using teacher entropy rather than student entropy (ablation)KD+CutMixPick · CIFAR100test loss: 1.0270 -> 0.9570Improved
Select augmented CutMix samples using student entropy rather than teacher entropy (ablation)KD+CutMixPick · ImageNettest loss: 1.1955 -> 1.2049Hurt
Use lower T. stddev augmentation schemesKD · CIFAR100 and Tiny ImageNetstudent test lossImproved
Use T. stddev to rank augmentation qualityKD · ImageNetstudent test lossNo clear effect
Add CutMix to other KD methodsAT, CC, SP, PKT, and VID · CIFAR100accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy