AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Knowledge Distillation Meets Self-Supervision

Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy

ECCV 2020 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 316 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
transfer incorrect self-supervision predictions with smallest error (k=75) in addition to correct ones (ablation)SSKD vgg13->vgg8 · CIFAR100top-1 accuracy (%): 74.19 -> 75.01Improved
transfer incorrect self-supervision predictions with smallest error (k=75) in addition to correct ones (ablation)SSKD resnet32x4->ShuffleV2 · CIFAR100top-1 accuracy (%): 77.65 -> 78.61Improved
transfer incorrect self-supervision predictions with smallest error (k=50) in addition to correct ones (ablation)SSKD wrn40-2->wrn16-2 · CIFAR100top-1 accuracy (%): 75.27 -> 75.97Improved
transfer all incorrect predictions (k=100) instead of filtering outliers (k=75) (ablation)SSKD vgg13->vgg8 · CIFAR100top-1 accuracy (%): 75.01 -> 74.77Hurt
transfer all incorrect predictions (k=100) instead of filtering outliers (k=75) (ablation)SSKD resnet32x4->ShuffleV2 · CIFAR100top-1 accuracy (%): 78.61 -> 77.97Hurt
use Contrastive instead of Exemplar self-supervision task (ablation)SSKD vgg13->vgg8 · CIFAR100top-1 accuracy (%): 74.57 -> 75.48Improved
use Rotation instead of Jigsaw task (ablation)SSKD vgg13->vgg8 · CIFAR100top-1 accuracy (%): 74.85 -> 75.01Improved
add self-supervision auxiliary losses L_T and L_ss to conventional KD (ablation)KD · CIFAR100 (four teacher-student pairs)top-1 accuracy (%)Improved
replace CRD by SSKD for cross-architecture distillationKD baselines · CIFAR100 (six different-architecture pairs)top-1 accuracy (%): +2.14% average over best competing methodsImproved
replace best competing method by SSKD for similar-architecture distillationKD baselines · CIFAR100 (five similar-architecture pairs)top-1 accuracy (%): +0.52% averageImproved
replace CRD+KD by SSKDResNet34->ResNet18 · ImageNettop-1 error (%): 28.62 -> 28.38Improved
reduce training data to 25% (few-shot)SSKD vgg13->vgg8 · CIFAR100 subsetstop-1 accuracy: ~7% absolute improvement over all competing methods at 25% dataImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy