Research map / Label-efficient and robust learning / Knowledge distillation
Self-supervised Knowledge Distillation Using Singular Value Decomposition
Seung Hyun Lee, Dae Ha Kim, Byung Cheol Song
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Yim et al. correlation knowledge with SVD+RBF distillation | VGG student · CIFAR-100 | accuracy (%): 64.54 -> 65.05 | Improved |
| replace Yim et al. knowledge with SVD+RBF distillation | MobileNet student · CIFAR-100 | accuracy (%): 56.53 -> 58.15 | Improved |
| replace Yim et al. knowledge with SVD+RBF distillation | ResNext student · CIFAR-100 | accuracy (%): 63.60 -> 65.43 | Improved |
| distill knowledge into a strided student with low-resolution feature maps | VGG student · CIFAR-100 (strided) | accuracy (%): 54.17 -> 61.15 | Improved |
| add one-stage self-supervised multi-task training instead of two-stage (ablation) | SVD+RBF distillation · CIFAR-100 (VGG) | accuracy (%): 65.05 -> 65.54 | Improved |
| add one-stage multi-task training instead of two-stage (ablation) | Yim et al. distillation · CIFAR-100 (VGG) | accuracy (%): 64.54 -> 64.89 | Improved |
| use SVD+RBF distillation instead of Yim et al. with 10% labels | VGG semi-supervised · CIFAR-100 (10% labels) | accuracy (%): 33.92 -> 49.68 | Improved |
| use SVD+RBF distillation instead of Yim et al. with 50% labels | VGG semi-supervised · CIFAR-100 (50% labels) | accuracy (%): 55.21 -> 60.01 | Improved |
| increase number of DFVs from 1 to 2 (ablation) | pooling student · CIFAR-100 | accuracy (%): 65.54 -> 66.33 | Improved |
| increase number of DFVs from 2 to 16 (ablation) | pooling student · CIFAR-100 | accuracy (%): 66.33 -> 65.15 | Hurt |
| increase number of DFVs from 1 to 16 (ablation) | strided student · CIFAR-100 | accuracy (%): 61.28 -> 62.00 | Improved |
Closest papers by meaning
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.90)
- Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (ICCV 2019 · similarity 0.89)
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.89)
- Self-Distillation: Towards Efficient and Compact Neural Networks (TPAMI 2021 · similarity 0.89)
- Data-Distortion Guided Self-Distillation for Deep Neural Networks (AAAI 2019 · similarity 0.89)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
