Research map / Label-efficient and robust learning / Knowledge distillation
Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard KD soft-label loss with weighted soft labels | Knowledge distillation · CIFAR-100, eight teacher-student pairs | Top-1 accuracy: 73.54 -> 74.48; 70.66 -> 72.15; 70.67 -> 72.19; 73.08 -> 74.12; 73.33 -> 76.05; 74.07 -> 75.46; 74.45 -> 75.93; 74.83 -> | Helped |
| Replace standard KD with weighted soft labels | ResNet-34 → ResNet-18 distillation · ImageNet validation | Top-1 accuracy: 70.67 -> 72.04 | Helped |
| Replace standard KD with weighted soft labels | ResNet-50 → MobileNet-v1 distillation · ImageNet validation | Top-1 accuracy: 70.49 -> 71.52 | Helped |
| Replace standard KD with weighted soft labels | BERT-12 → BERT-3 distillation · MultiNLI | Accuracy: 75.50 -> 76.28 | Helped |
| Mask KD gradients on the ground-truth-related logit (ablation) | Standard KD · CIFAR-100, WRN-40-2 → WRN-16-2, τ=4 | Top-1 accuracy: 74.92 -> 73.51 | Hurt |
| Exclude regularization samples from KD (ablation) | Standard KD · CIFAR-100, WRN-40-2 → WRN-16-2, τ=4 | Top-1 accuracy: 74.92 -> 74.59 | Hurt |
| Use KD only on regularization samples (ablation) | Standard KD · CIFAR-100, WRN-40-2 → WRN-16-2, τ=4 | Top-1 accuracy: 74.92 -> 73.86 | Hurt |
| Apply weighted soft labels to regularization samples only (ablation) | KD · CIFAR-100, WRN-40-2 → WRN-40-1 | Top-1 accuracy: 73.86 -> 74.46 | Helped |
| Apply weighted soft labels while excluding regularization samples (ablation) | KD · CIFAR-100, WRN-40-2 → WRN-40-1 | Top-1 accuracy: 74.59 -> 75.35 | Helped |
| Distill from a label-smoothed teacher instead of a teacher without label smoothing (ablation) | Standard KD, τ=4 · CIFAR-100 | Top-1 accuracy: 74.92 -> 74.59 | Hurt |
| Distill from a label-smoothed teacher instead of a teacher without label smoothing (ablation) | Weighted soft labels, τ=4 · CIFAR-100 | Top-1 accuracy: 75.78 -> 75.60 | Hurt |
| Replace proposed weighting function with scaled sigmoid weighting (ablation) | Weighted soft-label KD · CIFAR-100, WRN-40-2 → WRN-40-1, α=2.0 | Top-1 accuracy: 74.13 vs 74.38 | Hurt |
Closest papers by meaning
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.90)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.90)
- Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.88)
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.