Research map / Label-efficient and robust learning / Knowledge distillation
Improving Knowledge Distillation via Regularizing Feature Direction and Norm
Yuzhu Wang, Lechao Cheng, Manni Duan, Yongheng Wang, Zunlei Feng, Shu Kong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add feature direction regularization (cosine to teacher class-mean) to KD (ablation) | KD · CIFAR-100 (R50 -> MV2) | top-1 accuracy: 67.65 -> 69.18 | Improved |
| add feature norm regularization (SIFN) to KD (ablation) | KD · CIFAR-100 (R50 -> MV2) | top-1 accuracy: 67.65 -> 69.32 | Improved |
| use cosine rather than InfoNCE for direction regularization (ablation) | KD · CIFAR-100 (R56 -> R20) | top-1 accuracy: 70.73 -> 71.75 | Improved |
| naively add direction and norm regularizers together (ablation) | KD · CIFAR-100 (R50 -> MV2) | top-1 accuracy: 69.18 -> 68.62 | No clear effect |
| use unified dino-loss instead of separate direction plus norm terms (ablation) | KD · CIFAR-100 (R50 -> MV2) | top-1 accuracy: 69.07 -> 70.10 | Improved |
| use teacher class-mean features instead of classifier weights as direction targets (ablation) | dino-loss · CIFAR-100 (ResNet-50 -> MobileNet-V2) | top-1 accuracy: 69.76 -> 70.10 | Improved |
| match teacher classifier weights with L2 loss (ablation) | KD · CIFAR-100 (ResNet-50 -> MobileNet-V2) | top-1 accuracy: 67.65 -> 66.81 | Hurt |
| apply dino-loss at earlier layers instead of penultimate layer (ablation) | KD++ · CIFAR-100 (ResNet-50 -> MobileNet-V2) | top-1 accuracy: 70.10 -> 69.44 | Hurt |
| add dino-loss to KD | KD · ImageNet val (ResNet-34 -> ResNet-18) | top-1 accuracy: 70.66 -> 71.98 | Improved |
| add dino-loss to ReviewKD | ReviewKD · ImageNet val (ResNet-34 -> ResNet-18) | top-1 accuracy: 71.62 -> 71.64 | No clear effect |
| add dino-loss to KD while scaling teacher from ResNet-34 to ResNet-152 | KD · ImageNet val (ResNet-18 student) | top-1 accuracy: 71.99 -> 72.59 | Improved |
| scale teacher from ResNet-34 to ResNet-152 for plain KD | KD · ImageNet val (ResNet-18 student) | top-1 accuracy: 70.67 -> 71.13 | Mixed |
Closest papers by meaning
- Revisiting Knowledge Distillation via Label Smoothing Regularization (CVPR 2020 · similarity 0.93)
- Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation (ICLR 2023 · similarity 0.92)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.90)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
