Research map / Label-efficient and robust learning / Knowledge distillation
Switchable Online Knowledge Distillation
Biao Qian, Yang Wang, Hongzhi Yin, Richang Hong, Meng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the full SwitOKD (adaptive threshold, reciprocal teacher loss, switching) (ablation) | ResNet-32 / WRN-16-2 · CIFAR-100 (component ablation) | student top-1 accuracy (%): 72.91 -> 73.47 | Improved |
| use adaptive switching threshold instead of fixed delta 0.8 (ablation) | SwitOKD · CIFAR-100 (component ablation) | student top-1 accuracy (%): 73.00 -> 73.47 | Improved |
| use fixed delta 0.2 vs no switching (ablation) | SwitOKD · CIFAR-100 (component ablation) | student top-1 accuracy (%): 72.91 -> 72.92 | No clear effect |
| use fixed delta 0.6 vs no switching (ablation) | SwitOKD · CIFAR-100 (component ablation) | student top-1 accuracy (%): 72.91 -> 72.83 | No clear effect |
| use the proposed teacher loss instead of the alternative loss (case C) (ablation) | SwitOKD · CIFAR-100 (component ablation) | student top-1 accuracy (%): 72.72 -> 73.47 | Improved |
| set temperature tau to 2 instead of 1 (ablation) | SwitOKD MobileNetV2/WRN-16-2 · CIFAR-100 | student top-1 accuracy (%): 67.24 -> 67.80 | Improved |
| raise temperature tau to 10 from 2 (ablation) | SwitOKD MobileNetV2/WRN-16-2 · CIFAR-100 | student top-1 accuracy (%): 67.80 -> 65.64 | Hurt |
| use r = ratio with student plus teacher error in denominator for the pause threshold (ablation) | SwitOKD · CIFAR-100 | student top-1 accuracy (%): 71.89 -> 73.47 | Improved |
| SwitOKD instead of vanilla training on CIFAR-100 student | 0.5MobileNetV2 · CIFAR-100 | student top-1 accuracy (%): 60.07 -> 67.24 | Improved |
| SwitOKD instead of KDCL | 1.4MobileNetV2 student · Tiny-ImageNet | student top-1 accuracy (%): 57.79 -> 58.71 | Improved |
| SwitOKD instead of RCO offline | WRN-16-2 student · CIFAR-100 | student top-1 accuracy (%): 75.36 -> 75.95 | Improved |
| SwitOKD instead of KDCL for teacher | ResNet-34 teacher · ImageNet val | teacher top-1 accuracy (%): 73.70 -> 73.65 | Hurt |
Closest papers by meaning
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.89)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
