Research map / Label-efficient and robust learning / Knowledge distillation
Similarity-Preserving Knowledge Distillation
Frederick Tung, Greg Mori
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add similarity-preserving loss to student training | WideResNet-16-1 student, WRN-16-2 teacher · CIFAR-10 | test error (%): 8.74 -> 7.52 | Improved |
| add SP loss versus attention transfer | WideResNet-40-2 student, WRN-16-8 teacher · CIFAR-10 | test error (%): AT 4.47 vs SP 4.55 | Hurt |
| use activations from the end of each block instead of only the last conv layer (ablation) | SP distillation · CIFAR-10 | test error | No clear effect |
| use post-softmax scores instead of activations to compute similarity (ablation) | SP distillation · CIFAR-10 | test error | Hurt |
| add SP loss to fine-tuning of a pretrained student | MobileNet-0.25 (teacher MobileNet-0.5) · Describable Textures | test error (%): 42.45 -> 41.30 | Improved |
| add SP loss to fine-tuning of a pretrained student | MobileNetV2-0.35 (teacher MobileNetV2-1.0) · Describable Textures | test error (%): 41.25 -> 40.29 | Improved |
| use SP instead of attention transfer | MobileNet/MobileNetV2 students · Describable Textures | test error (%): SP won 19 of 30 MobileNet splits and 24 of 30 MobileNetV2 splits | Improved |
| add SP loss to student training | ShuffleNetV2-0.5 (teacher ShuffleNetV2-1.0) · CINIC-10 | test error (%): 20.09 -> 18.56 | Improved |
| combine SP with attention transfer | ShuffleNetV2-0.5 (teacher ShuffleNetV2-1.0) · CINIC-10 | test error (%): 18.50 -> 18.20 | Improved |
| combine SP with traditional KD | ShuffleNetV2-0.5 (teacher ShuffleNetV2-1.0) · CINIC-10 | test error (%): KD 18.62 -> KD+SP 18.35 | Mixed |
| combine SP with attention transfer | MobileNetV2-0.35 (teacher MobileNetV2-1.0) · CINIC-10 | test error (%): 16.10 -> 15.66 | Improved |
| vary SP weight gamma from 10 to 64,000 (ablation) | SP distillation · CINIC-10 | top-1 error | No clear effect |
Closest papers by meaning
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.92)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.92)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.91)
- Student Customized Knowledge Distillation: Bridging the Gap Between Student and Teacher (ICCV 2021 · similarity 0.91)
- Contrastive Representation Distillation (ICLR 2020 · similarity 0.91)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
