Research map / Label-efficient and robust learning / Knowledge distillation
Revisit the Essence of Distilling Knowledge through Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explains capacity mismatch in knowledge distillation via teacher calibration, showing KD performance correlates with teacher ECE and that ranking-based loss (DIST) is less sensitive to calibration.
Problem
Capacity mismatch in knowledge distillation: a larger, more accurate teacher can yield a weaker student, and existing explanations are fragmented.
Approach
Proposes a unifying calibration-based framework: teacher calibration (ECE) explains KD performance and the success of prior fixes (TAKD, early-stopped teachers); KL-based KD is sensitive to calibration, whereas ranking-based loss (DIST) is insensitive and can replace KL to let poorly calibrated large teachers teach better.
Key results
- Ranking-based DIST has a Spearman correlation of 0.46 (ResNet-56 teacher) and 1.82 (ResNet-110) between ECE and KD accuracy, versus -91.82 and -76.36 for vanilla KL-based KD, indicating much lower calibration sensitivity.
- On CIFAR-100, TAKD assistant ECE correlates with student accuracy: e.g., assistant WRN-28-2 (ECE 10.34%) gives 71.01% vs WRN-28-1 (ECE 11.08%) giving 70.36% with WRN-22-1 student.
- Posterior-trained model on a synthetic 10-class GMM task reaches 89.82% average test accuracy vs 89.13% with one-hot labels, with lower variance (3% vs 180%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace KL-based Vanilla KD with ranking-based DIST | ResNet-56 → ResNet-14 · CIFAR-100 | top-1 accuracy: 68.57 -> 68.93 | Helped |
| Replace KL-based Vanilla KD with ranking-based DIST | ResNet-110 → ResNet-14 · CIFAR-100 | top-1 accuracy: 68.42 -> 69.35 | Helped |
| Replace KL-based Vanilla KD with ranking-based DIST | WRN-28-1 → ResNet-14 · CIFAR-100 | top-1 accuracy: 67.76 -> 68.84 | Helped |
| Train with posterior distributions rather than one-hot labels | Classification model · 10-class classification test | accuracy: 89.13% -> 89.82% | Helped |
| Train with posterior distributions rather than one-hot labels | Classification model · 100-class classification test | accuracy: 94.00% -> 94.87% | Helped |
| Use an assistant model in TAKD, with assistant architecture varied (ablation) | TAKD · CIFAR-100 | KD performance: 67.96% to 69.00% | Mixed |
| Increase distillation temperature from 1.8 to 4.0 (ablation) | TAKD, WRN-16-3 → ResNet32 → WRN-16-1 · Tiny-ImageNet | KD performance: 49.66% -> 46.04% | Hurt |
| Increase distillation temperature from 1.8 to 4.0 (ablation) | TAKD, WRN-28-3 → WRN-28-2 → WRN-28-1 · Tiny-ImageNet | KD performance: 56.45% -> 52.87% | Hurt |
| Use teacher checkpoint 140 rather than 200 at temperature 1.0 (ablation) | ESKD, WRN-16-3 → WRN-16-1 · CIFAR-100 | KD performance: 67.31% -> 67.12% | Hurt |
| Use teacher checkpoint 140 rather than 200 at temperature 1.6 (ablation) | ESKD, WRN-16-3 → WRN-16-1 · CIFAR-100 | KD performance: 68.01% -> 68.43% | Helped |
| Use teacher checkpoint 130 rather than 200 at temperature 1.0 (ablation) | ESKD, WRN-28-3 → WRN-28-1 · CIFAR-100 | KD performance: 69.67% -> 71.09% | Helped |
| Train the teacher for fewer epochs (ablation) | WRN-16-3 → WRN-16-1 · CIFAR-100 | KD performance: 68.01% -> 68.62% | Helped |
Surprising findings
- Early-stopped (fewer-epoch) teachers can match or slightly exceed fully trained teachers in both accuracy-related KD performance and calibration.
- Ranking-based DIST is largely insensitive to teacher calibration, while KL-based KD shows strong negative correlation between ECE and student accuracy.
Limitations
- Calibration is measured with ECE only, and calibration is adjusted mainly via temperature scaling, a single intervention.
- Experiments are restricted to small image classification benchmarks (CIFAR, Tiny-ImageNet) with ResNet/WRN architectures.
- Correlation evidence is empirical; the causal link between calibration and capacity mismatch is argued rather than formally proven.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet synthetic GMM dataset
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: top-1 accuracy expected calibration error Spearman correlation between ECE and KD performance
Baselines: Vanilla KD DIST TAKD ESKD FitNet
Builds on: Vanilla KD TAKD DIST ESKD Residual KD
Closest papers by meaning
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.89)
- Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · similarity 0.88)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.88)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.