Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation Based on Transformed Teacher Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes transformed teacher matching (TTM), which drops temperature scaling on the student side of knowledge distillation, showing it equals KD plus a Rényi entropy regularizer, and a sample-adaptive variant WTTM that reaches state-of-the-art on CIFAR-100 and ImageNet.
Problem
Knowledge distillation conventionally applies temperature scaling to both teacher and student logits, and the role of this temperature on the student side is not well understood or justified.
Approach
Drop temperature scaling on the student side; reinterpret temperature scaling of the teacher as a power transform of its distribution, which shows TTM equals KD plus a Rényi entropy regularizer. Further weight each sample's distillation term by the power sum of the teacher distribution (WTTM).
Key results
- WTTM reaches 72.19% top-1 on ImageNet for ResNet-18 distilled from ResNet-34 (teacher 73.31), above KD at 70.66%.
- On CIFAR-100 same-architecture pairs, WTTM achieves 76.37% on WRN-40-2 to WRN-16-2 versus KD 74.92%.
- WTTM combined with ITRD reaches 77.36% on resnet32x4 to resnet8x4 on CIFAR-100, the best reported among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove student-side temperature scaling from KD, adding TTM’s Rényi entropy regularization | KD student · CIFAR-100, WRN-40-2 → WRN-16-2 | Top-1 accuracy (%): 74.92 -> 76.23 | Helped |
| Add sample-adaptive weighting to TTM to form WTTM (ablation) | TTM · CIFAR-100, WRN-40-2 → WRN-16-2 | Top-1 accuracy (%): 76.23 -> 76.37 | Helped |
| Add sample-adaptive weighting to TTM to form WTTM (ablation) | TTM · CIFAR-100, ResNet-50 → MobileNetV2 | Top-1 accuracy (%): 69.24 -> 69.59 | Helped |
| Add ITRD to WTTM | WTTM · CIFAR-100, WRN-40-2 → WRN-16-2 | Top-1 accuracy (%): 76.37 -> 76.65 | Helped |
| Add CRD to WTTM | WTTM · CIFAR-100, WRN-40-2 → WRN-16-2 | Top-1 accuracy (%): 76.37 -> 76.61 | Helped |
| Remove the cross-entropy loss from WTTM (ablation) | WTTM · CIFAR-100, WRN-40-2 → WRN-16-2 | Top-1 accuracy (%): 74.92 -> 75.11 | Mixed |
| Distill ResNet-18 from increasingly accurate ResNet teachers using WTTM | WTTM · ImageNet, ResNet-18 | Top-1 accuracy (%): 72.19 -> 72.26 -> 72.34 -> 72.39 | Helped |
| Add ViTKD to WTTM | WTTM · ImageNet, DeiT III-Small → DeiT-Tiny | Top-1 accuracy (%): 77.03 -> 78.04 | Helped |
Surprising findings
- Dropping the student-side temperature (TTM) consistently beats KD, and WTTM without the cross-entropy term still outperforms KD with it in most CIFAR-100 pairs.
Limitations
- Sample-adaptive weight U_{1/T}(p^t) is chosen heuristically; other weights are left unexplored.
- Only the power transform is studied; other point-wise transforms are left for future work.
- Evaluation limited to image classification on CIFAR-100 and ImageNet.
Experimental setup
Datasets: CIFAR-100 ImageNet
Benchmarks: CIFAR-100 ImageNet
Metrics: top-1 accuracy
Baselines: KD FitNet AT VID RKD PKT CRD DIST DKD ITRD SRRL ReviewKD
Builds on: knowledge distillation (Hinton et al.) confidence penalty label smoothing CRD ITRD
Closest papers by meaning
- Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
- Revisit the Essence of Distilling Knowledge through Calibration (ICML 2024 · similarity 0.88)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.