Research map / Label-efficient and robust learning / Knowledge distillation
Improved Knowledge Distillation via Teacher Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a CNN-4 teacher assistant between CNN-10 and CNN-2 | CNN knowledge distillation · CIFAR-10 | top-1 test accuracy: 72.57 -> 73.51 | Helped |
| Add a CNN-4 teacher assistant between CNN-10 and CNN-2 | CNN knowledge distillation · CIFAR-100 | top-1 test accuracy: 44.57 -> 44.92 | Helped |
| Add a ResNet-20 teacher assistant between ResNet-110 and ResNet-8 | ResNet knowledge distillation · CIFAR-10 | top-1 test accuracy: 88.65 -> 88.98 | Helped |
| Add a ResNet-20 teacher assistant between ResNet-110 and ResNet-8 | ResNet knowledge distillation · CIFAR-100 | top-1 test accuracy: 61.41 -> 61.82 | Helped |
| Add a ResNet-20 teacher assistant between ResNet-50 and ResNet-14 | ResNet knowledge distillation · ImageNet | top-1 test accuracy: 66.60 -> 67.36 | Helped |
| Use CNN-4 rather than CNN-8 as the teacher assistant (ablation) | CNN TAKD · CIFAR-10 | top-1 test accuracy: 72.75 -> 73.51 | Helped |
| Use CNN-4 rather than CNN-8 as the teacher assistant (ablation) | CNN TAKD · CIFAR-100 | top-1 test accuracy: 44.28 -> 44.92 | Helped |
| Use ResNet-14 rather than ResNet-56 as the teacher assistant (ablation) | ResNet TAKD · CIFAR-10 | top-1 test accuracy: 88.70 -> 88.98 | Helped |
| Use ResNet-20 rather than ResNet-56 as the teacher assistant (ablation) | ResNet TAKD · CIFAR-100 | top-1 test accuracy: 61.47 -> 61.82 | Helped |
| Train the teacher assistant by distillation rather than from scratch (ablation) | CNN TAKD · CIFAR-100 | student accuracy | Helped |
| Use a full path through available intermediate TAs rather than fewer-step paths (ablation) | CNN TAKD · CIFAR-100 | student accuracy | Helped |
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.92)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.