Research map / Label-efficient and robust learning / Knowledge distillation
A Good Teacher Adapts Their Knowledge for Distillation
Chengyao Qian, Trung Le, Mehrtash Harandi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add teacher fine-tuning (AID) before KD | vanilla KD, ResNet110x4 teacher · CIFAR-100, ResNet8 student | top-1 accuracy: 61.47 -> 63.44 | Improved |
| add teacher fine-tuning (AID) before KD | vanilla KD, ResNet110x4 teacher · CIFAR-100, ResNet14 student | top-1 accuracy: 66.96 -> 69.73 | Improved |
| add teacher fine-tuning (AID) before KD | vanilla KD, ResNet110x4 teacher · CIFAR-100, WRN-16-2 student | top-1 accuracy: 75.57 -> 76.68 | Improved |
| use larger teacher (ResNet32x4 instead of ResNet56) with vanilla KD | KD, ResNet20 student · CIFAR-100 | top-1 accuracy: 70.66 -> 69.83 | Hurt |
| use larger teacher (ResNet32x4 instead of ResNet56) with AID | AID, ResNet20 student · CIFAR-100 | top-1 accuracy: 72.50 -> 72.63 | Improved |
| distill only class-wise similarity term (ablation) | KD, ResNet20 student · CIFAR-100 (Figure 2) | top-1 accuracy | No clear effect |
| distill only intra-class distribution term (ablation) | KD, ResNet20 student · CIFAR-100 (Figure 2) | top-1 accuracy | Hurt |
| fine-tune teacher (ResNet32x4) with a different smaller student, ResNet20 student (ablation) | AID · CIFAR-100 | top-1 accuracy: 72.63 -> 72.43 across MFT choices | Improved |
| fine-tune teacher with larger model WRN-40-2 (ResNet20 student, ResNet32x4 teacher) (ablation) | AID · CIFAR-100 | top-1 accuracy: 70.04 | No clear effect |
| fine-tune ResNet32x4 teacher with ResNet20 instead of the target student for WRN-40-1 (ablation) | AID · CIFAR-100 | top-1 accuracy: 75.77 -> 75.08 | Hurt |
| use AID instead of retraining teacher/assistants (run-time) | TAKD · CIFAR-100 run-time | minutes: 699.8 -> 179.4 | Improved |
| add teacher fine-tuning (AID) | STD, ResNet110x4 teacher, ResNet8 · CIFAR-100 | top-1 accuracy: 59.43 -> 63.44 | Improved |
Closest papers by meaning
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.92)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.92)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.91)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.91)
- Revisit the Essence of Distilling Knowledge through Calibration (ICML 2024 · similarity 0.91)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
