Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation With the Reused Teacher Classifier
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse the teacher classifier and train the student encoder with feature alignment | Student networks distilled from larger teachers · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 76.23 $\pm$ 0.04 -> 78.08 $\pm$ 0.15 | Helped |
| Add SimKD to CMI data-free distillation | CMI · CIFAR-100 data-free KD, WRN-40-1 student | top-1 accuracy (%): 64.80 $\pm$ 0.35 -> 66.78 $\pm$ 0.29 | Helped |
| Replace the default three-layer projector with 1x1Conv (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 75.15 $\pm$ 0.27 | Hurt |
| Replace the default projector with two 1x1Conv layers (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 75.71 $\pm$ 0.33 | Hurt |
| Replace 3x3 convolution in the projector with depthwise convolution (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 77.76 $\pm$ 0.12 | Hurt |
| Align feature vectors rather than using last feature maps (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 76.03 $\pm$ 0.40 | Hurt |
| Move feature alignment from before to after the teacher classifier (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 77.09 $\pm$ 0.09 | Hurt |
| Add output feature loss to input feature alignment loss (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 77.88 $\pm$ 0.30 | Hurt |
| Reuse the teacher classifier plus its last building block (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 78.47 $\pm$ 0.08 | Helped |
| Reuse the teacher classifier plus its last two building blocks (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 78.88 $\pm$ 0.05 | Helped |
| Increase the projector reduction factor from r=2 to r=8 (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 76.73 $\pm$ 0.20 | Hurt |
| Jointly train the student encoder and classifier with KD and feature-alignment losses (ablation) | SimKD · CIFAR-100, ResNet-8x4 student and ResNet-32x4 teacher; reused teacher classifier | top-1 accuracy (%): 78.08 $\pm$ 0.15 -> 73.91 $\pm$ 0.12 (α=0.2) | Hurt |
Closest papers by meaning
- Knowledge Distillation from A Stronger Teacher (NeurIPS 2022 · similarity 0.92)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.91)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.90)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.89)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.