Research map / Label-efficient and robust learning / Knowledge distillation
Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Souvik Kundu, Qirui Sun, Yao Fu, Massoud Pedram, Peter Anthony Beerel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from nasty teacher at shallow auxiliary classifier with hybrid self-distillation (skeptical student) | ResNet18 student, nasty ResNet50 teacher · CIFAR-100 | accuracy (%): 72.28 -> 77.25 | Improved |
| skeptical student for MobileNetV2 from nasty ResNet50 | MobileNetV2 student · CIFAR-100 | accuracy (%): 7.72 -> 66.38 | Improved |
| skeptical student on Tiny-ImageNet from nasty teacher | ResNet18 student · Tiny-ImageNet | accuracy (%): 53.60 -> 65.76 | Improved |
| skeptical student from a normal teacher | ResNet50 student · CIFAR-100 | accuracy (%): 79.56 -> 80.66 | Improved |
| skeptical student from a normal teacher | ResNet18 student · CIFAR-10 (ResNet50 teacher) | accuracy (%): 95.43 -> 95.31 | No clear effect |
| ensemble ACs and final classifier when distilling from a nasty teacher (ablation) | skeptical student MobileNetV2 · CIFAR-100 | accuracy (%): 66.38 -> 62.93 | Hurt |
| ensemble ACs and final classifier when distilling from a normal teacher (ablation) | skeptical student ResNet50 · CIFAR-100 | accuracy (%): 80.66 -> 81.96 | Improved |
| distill from nasty teacher at the final classifier versus shallow AC1 (ablation) | ResNet18 student · CIFAR-100 | accuracy (%): 77.19 -> 72.47 | Hurt |
| skeptical student for data-free KD (grey-box, nasty ResNet50) | ResNet18 student · CIFAR-10 data-free | accuracy (%): 80.34 -> 86.14 | Improved |
| skeptical student for data-free KD (black-box, no AT loss, nasty ResNet50) | ResNet18 student · CIFAR-10 data-free | accuracy (%): 20.95 -> 79.93 | Improved |
| skeptical (instead of normal) primary student as teacher for secondary student | ResNet18 secondary student · CIFAR-100 | accuracy (%): 70.99 -> 79.22 | Improved |
Closest papers by meaning
- Efficient Verified Unlearning For Distillation (NeurIPS 2025 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.87)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.86)
- Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack (NeurIPS 2025 · similarity 0.86)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
