Research map / Label-efficient and robust learning / Knowledge distillation
Toward Student-oriented Teacher Network Training for Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoTeacher, a teacher training method that adds Lipschitz and consistency regularization to ERM so the teacher better approximates the true label distribution of its training data, improving student accuracy in knowledge distillation.
Problem
Teacher training for knowledge distillation is typically optimized for teacher accuracy, which does not necessarily yield the best student; the teacher trained to convergence overfits its soft predictions on reused distillation data.
Approach
Theoretically show that ERM minimizers approximate the true label distribution of training data under a mixed-feature data model when the feature extractor is Lipschitz and transformation-robust; then add a global Lipschitz regularizer and a temporal-ensembling consistency regularizer to teacher training.
Key results
- On CIFAR-100 (WRN40-2/WRN16-2), SoTeacher raises student accuracy from 74.87 to 75.39 while teacher accuracy drops from 76.38 to 74.95.
- On Tiny-ImageNet (ResNet34 to ResNet18), student top-1 improves from 66.19 to 66.83 with SoTeacher.
- On ImageNet (ResNet152 to ResNet18), student top-1 gains only marginally (71.30 to 71.45) while teacher top-1 drops from 77.87 to 77.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Lipschitz and consistency regularization during teacher training | Standard teacher training · CIFAR-100, three teacher-student pairs | student accuracy: 73.73\pm 0.13 -> 74.35\pm 0.23; 74.87\pm 0.45 -> 75.39\pm 0.23; 74.86\pm 0.18 -> 77.24\pm 0.09 | Helped |
| Add Lipschitz and consistency regularization during teacher training | Standard teacher training · Tiny-ImageNet, ResNet34/ResNet18 | student Top-1 accuracy: 66.19\pm 0.17 -> 66.83\pm 0.20 | Helped |
| Add Lipschitz and consistency regularization during teacher training | Standard teacher training · Tiny-ImageNet, ResNet34/ResNet18 | student Top-5 accuracy: 85.74\pm 0.21 -> 86.19\pm 0.22 | Helped |
| Add Lipschitz and consistency regularization during teacher training | Standard teacher training · ImageNet, ResNet152/ResNet18 | student Top-1 accuracy: 71.30 -> 71.45 | Helped |
| Add Lipschitz and consistency regularization during teacher training | Standard teacher training · CIFAR-100, three teacher architectures | teacher accuracy: 76.38\pm 0.13 -> 74.95\pm 0.28; 76.38\pm 0.13 -> 74.95\pm 0.28; 79.22\pm 0.03 -> 78.49\pm 0.09 | Hurt |
| Add consistency regularization (ablation) | SoTeacher · CIFAR-100, three teacher-student pairs | student accuracy: 74.34\pm 0.11 -> 74.35\pm 0.23; 75.20\pm 0.24 -> 75.39\pm 0.23; 76.52\pm 0.52 -> 77.24\pm 0.09 | Helped |
| Add Lipschitz regularization (ablation) | SoTeacher · CIFAR-100, three teacher-student pairs | student accuracy: 73.81\pm 0.15 -> 74.35\pm 0.23; 75.21\pm 0.13 -> 75.39\pm 0.23; 76.23\pm 0.18 -> 77.24\pm 0.09 | Helped |
| Add consistency regularization (ablation) | SoTeacher · Tiny-ImageNet, ResNet34/ResNet18 | student Top-1 accuracy: 66.39\pm 0.27 -> 66.83\pm 0.20 | Helped |
| Add consistency regularization (ablation) | SoTeacher · Tiny-ImageNet, ResNet34/ResNet18 | student Top-5 accuracy: 86.05\pm 0.17 -> 86.19\pm 0.22 | Helped |
| Add Lipschitz regularization (ablation) | SoTeacher · Tiny-ImageNet, ResNet34/ResNet18 | student Top-1 accuracy: 66.48\pm 0.43 -> 66.83\pm 0.20 | Helped |
| Add Lipschitz regularization (ablation) | SoTeacher · Tiny-ImageNet, ResNet34/ResNet18 | student Top-5 accuracy: 86.20\pm 0.40 -> 86.19\pm 0.22 | No clear effect |
Surprising findings
- Student accuracy improves even though the SoTeacher teacher is less accurate than the standard teacher.
- On ImageNet the teacher accuracy drops significantly under SoTeacher while student gains are minor.
Limitations
- Theory relies on a simplified mixed-feature (LDA-style) data distribution and an invariant or Lipschitz feature extractor assumption.
- Hyperparameters (lambda_LR, lambda_CR) were not extensively tuned; gains on ImageNet are minor.
- Experiments limited to image classification with CNN backbones on CIFAR-100, Tiny-ImageNet and ImageNet.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet ImageNet
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet
Metrics: top-1 accuracy top-5 accuracy ECE NLL student-teacher agreement (fidelity)
Baselines: Standard teacher training KD FitNets AT SP CC VID RKD PKT AB FT NST
Builds on: temporal ensembling knowledge distillation (Hinton et al.) Lipschitz regularization RCO
Closest papers by meaning
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.93)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.92)
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.91)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.