AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Switchable Online Knowledge Distillation

Biao Qian, Yang Wang, Hongzhi Yin, Richang Hong, Meng Wang

ECCV 2022 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 56 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add the full SwitOKD (adaptive threshold, reciprocal teacher loss, switching) (ablation)ResNet-32 / WRN-16-2 · CIFAR-100 (component ablation)student top-1 accuracy (%): 72.91 -> 73.47Improved
use adaptive switching threshold instead of fixed delta 0.8 (ablation)SwitOKD · CIFAR-100 (component ablation)student top-1 accuracy (%): 73.00 -> 73.47Improved
use fixed delta 0.2 vs no switching (ablation)SwitOKD · CIFAR-100 (component ablation)student top-1 accuracy (%): 72.91 -> 72.92No clear effect
use fixed delta 0.6 vs no switching (ablation)SwitOKD · CIFAR-100 (component ablation)student top-1 accuracy (%): 72.91 -> 72.83No clear effect
use the proposed teacher loss instead of the alternative loss (case C) (ablation)SwitOKD · CIFAR-100 (component ablation)student top-1 accuracy (%): 72.72 -> 73.47Improved
set temperature tau to 2 instead of 1 (ablation)SwitOKD MobileNetV2/WRN-16-2 · CIFAR-100student top-1 accuracy (%): 67.24 -> 67.80Improved
raise temperature tau to 10 from 2 (ablation)SwitOKD MobileNetV2/WRN-16-2 · CIFAR-100student top-1 accuracy (%): 67.80 -> 65.64Hurt
use r = ratio with student plus teacher error in denominator for the pause threshold (ablation)SwitOKD · CIFAR-100student top-1 accuracy (%): 71.89 -> 73.47Improved
SwitOKD instead of vanilla training on CIFAR-100 student0.5MobileNetV2 · CIFAR-100student top-1 accuracy (%): 60.07 -> 67.24Improved
SwitOKD instead of KDCL1.4MobileNetV2 student · Tiny-ImageNetstudent top-1 accuracy (%): 57.79 -> 58.71Improved
SwitOKD instead of RCO offlineWRN-16-2 student · CIFAR-100student top-1 accuracy (%): 75.36 -> 75.95Improved
SwitOKD instead of KDCL for teacherResNet-34 teacher · ImageNet valteacher top-1 accuracy (%): 73.70 -> 73.65Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy