Research map / Label-efficient and robust learning / Knowledge distillation
Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a knowledge distillation strategy that separates right from biased teacher knowledge, rectifies the teacher's wrong predictions, and uses a dynamically weighted easy-to-hard loss so the student can surpass the teacher on image classification and detection.
Problem
Standard knowledge distillation transfers the teacher's wrong predictions (bias) along with correct knowledge, which misleads the student and caps its performance below the teacher's.
Approach
Mask teacher predictions by correctness to split right knowledge from bias, then rectify biased teacher probabilities toward the true label via a weighted average and renormalization, and train with a loss whose weight shifts from easy (right-knowledge) to hard (rectified) targets over training (gamma = e/E).
Key results
- On CIFAR-100, ResNet-50 teacher to ResNet-18 student reaches 81.50% vs 71.56% for standard KD and a 79.34% teacher.
- On ImageNet 1K (ResNet-34 to ResNet-18), top-1 reaches 76.24% for ResNet-50 to MobileNet-V1 vs 72.56% best prior method; ResNet-34 to ResNet-18 reaches 73.38% vs 71.63%.
- On MS-COCO with Faster-RCNN-FPN, ResNet-101 to ResNet-18 student AP is 42.10 vs teacher 42.04 and KD 33.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bias elimination (ablation) | KD · CIFAR-100, ResNet-50 → ResNet-18 | accuracy: 71.56 -> 79.70 | Helped |
| add bias rectification (ablation) | KD · CIFAR-100, ResNet-50 → ResNet-18 | accuracy: 71.56 -> 73.86 | Helped |
| add bias elimination and bias rectification (ablation) | KD · CIFAR-100, ResNet-50 → ResNet-18 | accuracy: 71.56 -> 81.50 | Helped |
| add bias elimination (ablation) | KD · CIFAR-100, ResNet-50 → MobileNet-V2 | accuracy: 69.35 -> 79.85 | Helped |
| add bias rectification (ablation) | KD · CIFAR-100, ResNet-50 → MobileNet-V2 | accuracy: 69.35 -> 71.27 | Helped |
| add bias elimination and bias rectification (ablation) | KD · CIFAR-100, ResNet-50 → MobileNet-V2 | accuracy: 69.35 -> 80.45 | Helped |
| use the proposed bias rectification strategy | ResNet-50 → ResNet-18 KD · CIFAR-100 | accuracy: 73.97 -> 81.50 | Helped |
| use the proposed bias rectification strategy | ResNet-34 → ResNet-18 KD · ImageNet 1K val | top-1 accuracy: +1.75 | Helped |
| use the proposed bias rectification strategy | ResNet-50 → MobileNet-V1 KD · ImageNet 1K val | top-1 accuracy: +3.68 | Helped |
| use the proposed bias rectification strategy | KD · MS-COCO val2017, ResNet-101 → ResNet-18 | AP: 36.75 -> 42.10 | Helped |
Surprising findings
- Student models surpass their teachers on CIFAR-100, ImageNet and MS-COCO, which the authors claim is a first for KD.
- Reported ablation shows the bias rectification alone and elimination alone each improve over KD, with the combination best.
Limitations
- Gains on feature-based methods are smaller than on logit-based methods, attributed to misleading intermediate-layer information.
- Code is promised only after acceptance; no release at submission time.
- Evaluated only on classification and detection; the bias rectification is derived for a binary-style two-class formulation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet 1K MS-COCO
Benchmarks: CIFAR-10 CIFAR-100 ImageNet 1K MS-COCO
Metrics: top-1 accuracy top-5 accuracy AP AP50 AP70
Baselines: KD FitNet RKD CRD OFD ReviewKD InDistill DKD CTKD AT
Builds on: knowledge distillation DKD CTKD FitNet
Closest papers by meaning
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.91)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.91)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.90)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.90)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.90)
- Revisit the Essence of Distilling Knowledge through Calibration (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.