Research map / Label-efficient and robust learning / Knowledge distillation
Self-Knowledge Distillation With Progressive Refinement of Targets
Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, Sangheum Hwang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PS-KD | ResNet-18 · CIFAR-100 | top-1 error (%): 24.18 -> 20.82 | Improved |
| add PS-KD | PyramidNet · CIFAR-100 | top-1 error (%): 16.80 -> 15.49 | Improved |
| add PS-KD to Cutout | PyramidNet + Cutout · CIFAR-100 | top-1 error (%): 16.05 -> 14.82 | Improved |
| add PS-KD to CutMix | PyramidNet + CutMix · CIFAR-100 | top-1 error (%): 15.62 -> 15.03 | Improved |
| add PS-KD to CutMix + ShakeDrop | PyramidNet + CutMix + SD · CIFAR-100 | top-1 error (%): 14.07 -> 13.59 | Improved |
| add PS-KD | ResNet-152 · ImageNet val | top-1 error (%): 22.19 -> 21.41 | Improved |
| add PS-KD to CutMix | ResNet-152 + CutMix · ImageNet val | top-1 error (%): 21.04 -> 20.76 | Improved |
| add TF-KD to CutMix | ResNet-152 + CutMix · ImageNet val | top-1 error (%): 21.04 -> 22.00 | Hurt |
| add label smoothing to Cutout | PyramidNet + Cutout · CIFAR-100 | top-1 error (%): 16.05 -> 17.15 | Hurt |
| add CS-KD | PyramidNet · CIFAR-100 | top-1 error (%): 16.80 -> 18.31 | Hurt |
| add PS-KD | ResNeXt-29 · CIFAR-100 | ECE (%): 4.17 -> 9.15 | Hurt |
| use PS-KD-pretrained backbone | Faster R-CNN ResNet-152 · PASCAL VOC 2007 test | mAP (%): 78.26 -> 79.50 | Improved |
Closest papers by meaning
- Self-Distillation From the Last Mini-Batch for Consistency Regularization (CVPR 2022 · similarity 0.92)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.89)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
