Research map / Label-efficient and robust learning / Knowledge distillation
Self-Distillation as Instance-Specific Label Smoothing
Zhilu Zhang, Mert Sabuncu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase teacher temperature | Self-distillation with ResNet-34 · CIFAR-100 | test accuracy | Improved |
| Increase teacher temperature beyond the point where confidence diversity plateaus | Self-distillation with ResNet-34 · CIFAR-100 | NLL | Hurt |
| Use self-distillation instead of label smoothing at matched average smoothing | ResNet and DenseNet · CIFAR-100, CUB-200, and Tiny-ImageNet | test accuracy | Improved |
| Use Beta smoothing instead of label smoothing | ResNet and DenseNet · CUB-200 and Tiny-ImageNet; CIFAR-100 ResNet is an exception | test accuracy | Mixed |
| Use self-distillation instead of label smoothing | ResNet and DenseNet · CIFAR-100, CUB-200, and Tiny-ImageNet | expected calibration error | Improved |
| Use Beta smoothing instead of label smoothing | ResNet and DenseNet · CIFAR-100, CUB-200, and Tiny-ImageNet | expected calibration error | Improved |
| Use EMA confidence rankings to assign Beta-smoothed labels rather than assigning random Beta labels (ablation) | Beta smoothing · CIFAR-100, CUB-200, and Tiny-ImageNet | test accuracy | Improved |
| Use EMA confidence rankings to assign Beta-smoothed labels rather than assigning random Beta labels (ablation) | Beta smoothing · CIFAR-100, CUB-200, and Tiny-ImageNet | expected calibration error | Improved |
| Use Beta smoothing instead of explicit self-training with EMA predictions (ablation) | ResNet · CIFAR-100, CUB-200, and Tiny-ImageNet | test accuracy | Improved |
| Use Beta smoothing instead of explicit self-training with EMA predictions (ablation) | ResNet · CIFAR-100, CUB-200, and Tiny-ImageNet | expected calibration error | Improved |
| Use a better teacher to provide instance-specific priors (ablation) | Cross-distillation · CIFAR-100, CUB-200, and Tiny-ImageNet | test accuracy | Improved |
| Prune 50% of classes from teacher predictions during distillation (ablation) | Pruned distillation · CIFAR-100 and Tiny-ImageNet | test accuracy | No clear effect |
Closest papers by meaning
- Understanding Self-Distillation in the Presence of Label Noise (ICML 2023 · similarity 0.90)
- Self-Distillation From the Last Mini-Batch for Consistency Regularization (CVPR 2022 · similarity 0.90)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.88)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.88)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.86)
- Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
