Research map / Label-efficient and robust learning / Knowledge distillation
Understanding the Effects of Projectors in Knowledge Distillation
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, Branislav Kusý, Zi Helen Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a single projector to feature distillation (ablation) | feature distillation ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 73.66 -> 75.14 | Improved |
| ensemble three projectors instead of one (ablation) | projector ensemble ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 75.14 -> 76.08 | Improved |
| ensemble four projectors instead of three (ablation) | projector ensemble ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 76.08 -> 75.93 | No clear effect |
| deepen projector to 4-layer MLP (ablation) | projector ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 75.14 -> 74.30 | Hurt |
| deepen projector to 4-layer MLP (ablation) | projector VGG13-VGG8 · CIFAR-100 | top-1 accuracy: 73.84 -> 72.73 | Hurt |
| widen hidden layers of 2-layer MLP projector (ablation) | 2L-MLP projector VGG13-VGG8 · CIFAR-100 | top-1 accuracy: 73.31 -> 73.37 | No clear effect |
| add ReLU activation to projector ensemble (ablation) | projector ensemble ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 75.04 -> 76.08 | Improved |
| replace ReLU with GELU (ablation) | projector ensemble ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 76.08 -> 76.32 | Improved |
| use mixed instead of default initialization (ablation) | projector ensemble ResNet32x4-ResNet8x4 · CIFAR-100 | top-1 accuracy: 76.08 -> 76.27 | Improved |
| use orthogonal instead of default initialization (ablation) | projector ensemble VGG13-VGG8 · CIFAR-100 | top-1 accuracy: 74.35 -> 73.89 | Hurt |
| add projector to feature distillation on ImageNet (ablation) | projector ResNet34-ResNet18 · ImageNet | top-1 accuracy: 71.63 -> 71.94 | Improved |
| ensemble projectors for ResNet50-MobileNet (ablation) | projector ensemble · ImageNet | top-1 accuracy: 72.75 -> 73.29 | Improved |
Closest papers by meaning
- Improved Feature Distillation via Projector Ensemble (NeurIPS 2022 · similarity 0.94)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.90)
- Knowledge Distillation With the Reused Teacher Classifier (CVPR 2022 · similarity 0.87)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.85)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.85)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
