Research map / Label-efficient and robust learning / Knowledge distillation
Distilling the Undistillable: Learning from a Nasty Teacher
Surgan Jandial, Yash Khasbage, Arghya Pal, Vineeth N Balasubramanian, Balaji Krishnamurthy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use high-temperature softened teacher output composed with one-hot (HTC) instead of plain Nasty KD | Nasty KD · CIFAR-100 ResNet18->MobileNetV2 | student accuracy: 3.15 -> 71.01 | Improved |
| use sequence of contrastive models ensemble (SCM) instead of Nasty KD | Nasty KD · TinyImageNet ResNet18->MobileNetV2 | student accuracy: 0.85 -> 61.01 | Improved |
| use SCM instead of Skeptical | Skeptical · TinyImageNet ResNet18->MobileNetV2 | student accuracy: 47.37 -> 61.01 | Improved |
| use SCM sequence length 4 instead of 2 (ablation) | SCM · CIFAR-100 ShuffleNetV2 | student accuracy: 74.45 -> 75.23 | Improved |
| use SCM sequence length 4 instead of 2 (ablation) | SCM · CIFAR-100 CNN | student accuracy: 60.99 -> 61.31 | Improved |
| generate SCM sequence with ShuffleNetV2 instead of ResNet18 architecture (ablation) | SCM · CIFAR-100 ShuffleNetV2 | student accuracy: 63.67 -> 72.71 | Improved |
| replace Nasty teacher by SCM ensemble (ablation) | Nasty Teacher · CIFAR-100 ResNet18 | KL divergence to original teacher: 14.6363 -> 5.8398 | Improved |
| apply HTC when no labels are available | Nasty KD · CIFAR-100 ShuffleNetV2 | student accuracy: 64.41 -> 74.17 | Improved |
| apply HTC in data-free distillation | Nasty KD · CIFAR-10 MobileNetV2 | student accuracy: 41.79 -> 52.96 | Improved |
| use only 10% of the data with HTC | Nasty KD · CIFAR | student accuracy | Improved |
| use intermediate SCM sequence model as teacher (improved defense) | Nasty KD · CIFAR-10 MobileNetV2 | student accuracy: 31.73 -> 22.12 | Improved |
| use early-stopped Nasty teacher matched in accuracy instead of SCM-based defense (ablation) | Ours defense · CIFAR-10 ShuffleNetV2 | student accuracy: 73.23 -> 87.81 | Hurt |
Closest papers by meaning
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.86)
- Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack (NeurIPS 2025 · similarity 0.85)
- Efficient Verified Unlearning For Distillation (NeurIPS 2025 · similarity 0.84)
- On Teacher Hacking in Language Model Distillation (ICML 2025 · similarity 0.84)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.83)
- Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
