Research map / Label-efficient and robust learning / Knowledge distillation
Learning Student-Friendly Teacher Networks for Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train teacher with student branches (SFTN) instead of standard teacher | KD · CIFAR-100 resnet32x4->resnet8x4 | student accuracy: 73.39 -> 76.09 | Helped |
| train teacher with student branches (SFTN) | KD · CIFAR-100 resnet32x4->ShuffleV1 | student accuracy: 74.26 -> 77.93 | Helped |
| train teacher with student branches (SFTN) | KD · ImageNet ResNet50->ResNet34 top-1 | student accuracy: 73.55 -> 74.14 | Helped |
| SFTN teacher vs collaborative learning DML | KD · CIFAR-100 resnet32x4->ShuffleV1 | student accuracy: 73.37 -> 77.93 | Helped |
| SFTN teacher vs curriculum RCO | KD · CIFAR-100 resnet32x4->ShuffleV1 | student accuracy: 76.62 -> 77.93 | Helped |
| train SFTN four times longer (SFTN-4) (ablation) | SFTN · CIFAR-100 resnet32x4->ShuffleV1 | student accuracy: 77.93 -> 79.11 | Helped |
| increase KL weight lambda_KL from 1 to 3 (ablation) | SFTN · CIFAR-100 KD average student accuracy | accuracy: 76.07 -> 76.71 | Helped |
| increase KL weight lambda_KL to 10 (ablation) | SFTN · CIFAR-100 KD average student accuracy | accuracy: 76.71 -> 74.87 | Hurt |
| vary KL temperature tau~ from 1 to 20 (ablation) | SFTN · CIFAR-100 student accuracy | accuracy: 75.58 -> 74.95 | No clear effect |
| fine-tune pretrained teacher with student branches instead of training from scratch (ablation) | SFTN · CIFAR-100 resnet32x4->ShuffleV2 | average student accuracy: 78.81 -> 78.71 | No clear effect |
| use student branch with smaller capacity than student (ablation) | SFTN · CIFAR-100 WRN40-2/WRN16-2 | average student accuracy: 76.66 -> 74.91 | Hurt |
| SFTN teacher vs standard teacher on corrupted data | KD · CIFAR-100-C intensity 5 | accuracy: 32.97 -> 33.15 | Helped |
Closest papers by meaning
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.92)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.92)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.91)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.