Research map / Label-efficient and robust learning / Knowledge distillation
Student Customized Knowledge Distillation: Bridging the Gap Between Student and Teacher
Yichen Zhu, Yi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gradient-similarity gating of KD losses | baseline KD, ResNet18 student · CIFAR100 | top-1 accuracy (%): 71.02 -> 71.73 | Improved |
| add gradient-similarity gating of KD losses | baseline KD, ResNet50 student · CIFAR100 | top-1 accuracy (%): 74.95 -> 75.64 | Improved |
| use SCKD instead of early-stopping KD (ESKD) | ESKD, ResNet18 student · ImageNet | top-1 accuracy (%): 70.7 -> 71.3 | Improved |
| use SCKD instead of teacher assistant KD (TAKD) | TAKD, ShuffleNetV2 student · ImageNet | top-1 accuracy (%): 70.4 -> 71.3 | Improved |
| add SCKD gating to TOFD | TOFD, ResNet18 · CIFAR100 | accuracy (%): 82.92 -> 84.16 | Improved |
| add SCKD gating to TOFD | TOFD, SENet18 · CIFAR100 | accuracy (%): 84.44 -> 85.49 | Improved |
| add SCKD gating to TOFD | TOFD, ShuffleNetV2 · CIFAR100 | accuracy (%): 76.68 -> 77.58 | Improved |
| add SCKD gating to FBOD | FBOD, Faster RCNN ResNet18 · MS COCO | mAP: 37.0 -> 37.5 | Improved |
| add SCKD gating to FBOD | FBOD, RetinaNet ResNet18 · MS COCO | mAP: 35.9 -> 36.5 | Improved |
| add SCKD gating to IFVD | IFVD, PSPNet-ResNet18 (0.5) · Cityscapes val | mIoU (%): 63.35 -> 65.10 | Improved |
| add SCKD gating to IFVD | IFVD, PSPNet-EfficientNetB0 · Cityscapes test | mIoU (%): 62.52 -> 63.08 | Improved |
| increase teacher size with SCKD versus conventional KD (ablation) | plain CNN student · CIFAR10 and CIFAR100 | student gain | Improved |
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.93)
- Distilling Knowledge via Knowledge Review (CVPR 2021 · similarity 0.93)
- Knowledge Distillation from A Stronger Teacher (NeurIPS 2022 · similarity 0.92)
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.92)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.91)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
