Research map / Label-efficient and robust learning / Knowledge distillation
Understanding the Role of the Projector in Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits knowledge distillation as function matching and metric learning, showing that a linear projector with batch normalisation and a LogSum distance implicitly captures relational information and yields a cheap, competitive distillation recipe.
Problem
Knowledge distillation designs (distance metric, normalisation, projector) are ad hoc and computationally heavy, with relational structures or memory banks needed, and there is little theoretical understanding of why they work; large capacity gaps also hurt representation matching.
Approach
A linear bias-free projector trained jointly with the student, combined with batch normalisation of representations and a LogSum distance, implicitly encodes relational information from past samples, removing the need for explicit correlation matrices or memory banks; LogSum softens matches to handle large capacity gaps.
Key results
- Achieves 77.2% top-1 on ImageNet with DeiT-Ti distilled from RegNetY-160, versus 76.6% for the best prior method with 1000 epochs.
- On CIFAR-100, reaches 79.06 top-1 for ShuffleV2 student (R32x4 teacher) and 79.22 for ShuffleV1 with WRN40-2 teacher, among the best reported.
- Batch normalisation of representations outperforms no normalisation and L2 normalisation across architecture pairs (e.g., 56.09 vs 50.89 for RegNet to MobileNetV2 on ImageNet 20% subset).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use batch normalization instead of L2 normalization (ablation) | Feature distillation · ImageNet-1K 20% subset, RegNet → MBv2 | top-1 accuracy: 52.91 -> 56.09 | Helped |
| Use batch normalization instead of L2 normalization (ablation) | Feature distillation · ImageNet-1K 20% subset, ViT → MBv3 | top-1 accuracy: 54.65 -> 59.28 | Helped |
| Use batch normalization instead of L2 normalization (ablation) | Feature distillation · ImageNet-1K 20% subset, ConvNeXt → EffNet-b0 | top-1 accuracy: 64.23 -> 67.95 | Helped |
| Add LogSum to the distillation loss (ablation) | Feature distillation · ImageNet-1K 20% subset, ViT → MBv3 | top-1 accuracy: 59.28 -> 59.80 | Helped |
| Add LogSum to the distillation loss (ablation) | Feature distillation · ImageNet-1K 20% subset, ConvNeXt → EffNet-b0 | top-1 accuracy: 67.95 -> 68.51 | Helped |
| Add LogSum to the distillation loss (ablation) | Feature distillation · ImageNet-1K, ResNet50 → ResNet18 | top-1 accuracy: 70.03 -> 71.29 | Helped |
| Vary LogSum α across the tested values (ablation) | Feature distillation · ImageNet-1K, ResNet50 → ResNet18 | top-1 accuracy: 61.74; 62.23; 63.10; 63.32; 63.40 | Mixed |
| Replace the linear projector with a larger projector network (ablation) | Feature distillation · Image classification | student performance | No clear effect |
| Apply the proposed recipe instead of DearKD (1000 epochs) | DeiT-Ti data-efficient training · ImageNet-1K | acc@1: 77.0 -> 77.2 | Helped |
| Apply the proposed recipe instead of DearKD (1000 epochs) | DeiT-S data-efficient training · ImageNet-1K | acc@1: 82.8 -> 82.1 | Hurt |
| Apply the proposed recipe instead of CRD | CIFAR100, WRN40-2 → WRN16-2 with RandAugment · CIFAR100 | top-1 accuracy: 76.04 -> 76.14 | Helped |
| Apply the proposed recipe instead of ReviewKD | Faster R-CNN with MV2 · COCO2017 | mAP: 33.71 -> 32.92 | Hurt |
Surprising findings
- A trivially larger projector network does not improve student performance because it learns to decorrelate input-output features, indicating a trade-off between projector capacity and distillation efficacy.
- Even a single linear projector gives significant accuracy gains even when student and teacher dimensions already match.
Limitations
- Theoretical analysis is limited to a simple L2 loss with linear bias-free projector; other normalisation schemes are only analysed empirically.
- Gains are smaller when the capacity gap is small (e.g., ImageNet CNN pairs and DeiT-S vs. RegNet teacher), and ResNet/ImageNet results are not state-of-the-art for all settings.
- Some baselines (e.g., ICKD, SimKD) were omitted because they modify training settings or architectures.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K COCO2017
Benchmarks: CIFAR-100 ImageNet-1K COCO2017
Metrics: top-1 accuracy top-5 error mAP AP50 translational equivariance measure
Baselines: KD FitNet AT CRD SSKD DKD ReviewKD DeiT CivT DearKD USKD FunMatch
Builds on: CRD ReviewKD SSKD Co-Advice torchdistill
Closest papers by meaning
- Improved Feature Distillation via Projector Ensemble (NeurIPS 2022 · similarity 0.91)
- Knowledge Distillation from A Stronger Teacher (NeurIPS 2022 · similarity 0.89)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.89)
- Supervision Complexity and its Role in Knowledge Distillation (ICLR 2023 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.