Research map / Label-efficient and robust learning / Knowledge distillation
Logit Standardization in Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a plug-and-play Z-score logit standardization (with weighted standard deviation as temperature) applied before softmax in logit-based knowledge distillation, so students match teacher logit relations rather than magnitudes.
Problem
Shared temperature in logit-based knowledge distillation implicitly forces the student logits to match the teacher's logit range and variance, which is hard for small students with large teachers and can make KL loss a misleading measure of student performance.
Approach
Derives via entropy maximization that teacher and student temperatures are independent and sample-wise; then sets the temperature to the weighted standard deviation of logits and applies a Z-score standardization before softmax and KL, so only logit relations (rank and relative spacing) are transferred.
Key results
- On CIFAR-100 with ResNet32x4 teacher and ResNet8x4 student, KD+Ours reaches 76.62% top-1 versus 73.33% for vanilla KD.
- On ImageNet with ResNet50 teacher and MobileNetV1 student, KD+Ours achieves 72.18% top-1 versus 70.50% for vanilla KD (+1.68).
- Applying the standardization to DKD and MLKD yields consistent gains, e.g., MLKD+Ours on ResNet32x4/ResNet8x4 reaches 78.28% vs 77.08% (CIFAR-100, identical-architecture setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Z-score logit standardization to logit-based KD methods | KD, CTKD, DKD, and MLKD · CIFAR-100 val | top-1 accuracy: +1.11, +0.81, +0.30, and +0.32 | Helped |
| Add Z-score logit standardization to logit-based KD methods | KD, CTKD, DKD, and MLKD · CIFAR-100 val | top-1 accuracy: +3.29, +3.28, +0.69, and +1.20 | Helped |
| Add Z-score logit standardization to logit-based KD methods | KD, CTKD, DKD, and MLKD · ImageNet val | top-1 accuracy: +0.39, +0.43, +0.18, and +0.18 | Helped |
| Add Z-score logit standardization to logit-based KD methods | KD, CTKD, DKD, and MLKD · ImageNet val | top-5 accuracy: +0.24, +0.19, +0.17, and +0.19 | Helped |
| Use full Z-score standardization instead of original logits (ablation) | KD · CIFAR-100 val | top-1 accuracy: 73.60 -> 74.14; 74.38 -> 76.11; 74.45 -> 76.56; 73.33 -> 76.62; 68.29 -> 76.56; 65.34 -> 76.61; 63.45 -> 76.33 | Helped |
| Center logits without standard-deviation normalization (ablation) | Z-score logit standardization · CIFAR-100 val | top-1 accuracy: 73.37, 74.33, 74.82, 73.94, 71.56, 62.01, 61.31 | Mixed |
| Normalize logits by standard deviation without centering (ablation) | Z-score logit standardization · CIFAR-100 val | top-1 accuracy: 73.79, 75.86, 76.44, 76.30, 76.49, 76.42, 76.18 | Helped |
| Vary the KD-loss weight λKD from 0.9 to 18.0 (ablation) | KD without logit standardization · CIFAR-100 val | top-1 accuracy: 74.45 -> 63.45 | Hurt |
| Vary the KD-loss weight λKD from 0.9 to 18.0 (ablation) | Z-score logit standardization · CIFAR-100 val | top-1 accuracy: 74.14, 76.11, 76.56, 76.62, 76.56, 76.61, 76.33 | No clear effect |
Surprising findings
- Under the shared-temperature setting, a student with lower KL loss can still predict the wrong class, so the KL loss can be misleading as a measure of student performance (toy case).
- Vanilla KD with a larger KD loss weight does not give considerable gains, while the Z-score pre-process yields significant improvements at large weights.
Limitations
- Gradient through the Z-score requires a larger KD loss weight to compensate, adding a hyperparameter to tune.
- Experiments are limited to image classification on CIFAR-100 and ImageNet in the main text; other tasks are only in the supplement.
Experimental setup
Datasets: CIFAR-100 ImageNet
Benchmarks: CIFAR-100 ImageNet
Metrics: top-1 accuracy top-5 accuracy
Baselines: KD CTKD DKD MLKD FitNet AT RKD CRD OFD ReviewKD SimKD CAT-KD
Builds on: Knowledge distillation (Hinton et al.) CTKD DKD MLKD ATKD
Closest papers by meaning
- Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · similarity 0.89)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.89)
- Scaled Decoupled Distillation (CVPR 2024 · similarity 0.89)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.88)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.