Research map / Label-efficient and robust learning / Knowledge distillation
Revisiting Knowledge Distillation via Label Smoothing Regularization
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a stronger teacher with the model's own pretrained predictions | MobileNetV2 · CIFAR100 | accuracy: 68.38 -> 70.96 | Helped |
| Use a manually designed regularization distribution instead of a trained teacher | ResNet50 · ImageNet-2012 | Top1 accuracy: 75.77 -> 76.42 | Helped |
| Use Tf-KD_reg rather than label smoothing | ResNet50 · ImageNet-2012 | Top1 accuracy: 76.38 -> 76.42 | Helped |
| Use label smoothing regularization | DenseNet121 · ImageNet-2012 | Top1 accuracy: 75.28 -> 75.24 | Hurt |
| Self-distill the model using its pretrained predictions | ResNet18 · ImageNet-2012 | Top1 accuracy: 69.84 -> 70.42 | Helped |
| Replace strong-teacher KD with self-distillation | ResNet18 · ImageNet-2012 | Top1 accuracy: 70.42 vs 70.40 | No clear effect |
| Let a weaker MobileNetV2 teach ResNet18 | ResNet18 · CIFAR100 | accuracy: 75.87 -> 77.28±0.28 | Helped |
| Distill from a one-epoch, poorly trained ResNet18 | MobileNetV2 · CIFAR100 | accuracy: 68.38 -> 70.65±0.35 | Helped |
| Distill from a poorly trained ResNeXt29 | MobileNetV2 · CIFAR100 | accuracy: 68.38 -> 71.52±0.27 | Helped |
| Use self-distillation rather than a strong teacher | ResNeXt29 · CIFAR100 | accuracy: +1.05 vs +1.12 | No clear effect |
| Use manually designed regularization rather than self-distillation | ResNeXt101 · ImageNet-2012 | Top1 accuracy: 79.56 vs 79.76 | Mixed |
| Use self-distillation rather than manually designed regularization | MobileNetV2 · Tiny-ImageNet | accuracy: 56.77 vs 56.47 | Mixed |
Closest papers by meaning
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.90)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.89)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.