Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation Meets Self-Supervision
Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer incorrect self-supervision predictions with smallest error (k=75) in addition to correct ones (ablation) | SSKD vgg13->vgg8 · CIFAR100 | top-1 accuracy (%): 74.19 -> 75.01 | Improved |
| transfer incorrect self-supervision predictions with smallest error (k=75) in addition to correct ones (ablation) | SSKD resnet32x4->ShuffleV2 · CIFAR100 | top-1 accuracy (%): 77.65 -> 78.61 | Improved |
| transfer incorrect self-supervision predictions with smallest error (k=50) in addition to correct ones (ablation) | SSKD wrn40-2->wrn16-2 · CIFAR100 | top-1 accuracy (%): 75.27 -> 75.97 | Improved |
| transfer all incorrect predictions (k=100) instead of filtering outliers (k=75) (ablation) | SSKD vgg13->vgg8 · CIFAR100 | top-1 accuracy (%): 75.01 -> 74.77 | Hurt |
| transfer all incorrect predictions (k=100) instead of filtering outliers (k=75) (ablation) | SSKD resnet32x4->ShuffleV2 · CIFAR100 | top-1 accuracy (%): 78.61 -> 77.97 | Hurt |
| use Contrastive instead of Exemplar self-supervision task (ablation) | SSKD vgg13->vgg8 · CIFAR100 | top-1 accuracy (%): 74.57 -> 75.48 | Improved |
| use Rotation instead of Jigsaw task (ablation) | SSKD vgg13->vgg8 · CIFAR100 | top-1 accuracy (%): 74.85 -> 75.01 | Improved |
| add self-supervision auxiliary losses L_T and L_ss to conventional KD (ablation) | KD · CIFAR100 (four teacher-student pairs) | top-1 accuracy (%) | Improved |
| replace CRD by SSKD for cross-architecture distillation | KD baselines · CIFAR100 (six different-architecture pairs) | top-1 accuracy (%): +2.14% average over best competing methods | Improved |
| replace best competing method by SSKD for similar-architecture distillation | KD baselines · CIFAR100 (five similar-architecture pairs) | top-1 accuracy (%): +0.52% average | Improved |
| replace CRD+KD by SSKD | ResNet34->ResNet18 · ImageNet | top-1 error (%): 28.62 -> 28.38 | Improved |
| reduce training data to 25% (few-shot) | SSKD vgg13->vgg8 · CIFAR100 subsets | top-1 accuracy: ~7% absolute improvement over all competing methods at 25% data | Improved |
Closest papers by meaning
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.91)
- Multi-target Knowledge Distillation via Student Self-reflection (IJCV 2023 · similarity 0.91)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.91)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.91)
- Contrastive Representation Distillation (ICLR 2020 · similarity 0.91)
- Similarity-Preserving Knowledge Distillation (ICCV 2019 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
