Research map / Label-efficient and robust learning / Knowledge distillation
Feature Normalized Knowledge Distillation for Image Classification
Kunran Xu, Lai Rui, Yishi Li, Lin Gu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace unified temperature with feature-norm-based sample-specific factor (KD-fn) | KD, Resnet20 student · Cifar-100 | accuracy (%): 80.34 -> 81.19 | Improved |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Resnet56 student · Cifar-100 | accuracy (%): 81.82 -> 82.23 | Improved |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Resnet20 student · Cifar-10 | accuracy (%): 92.86 -> 92.67 | No clear effect |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Resnet18 student · CUB-200-2011 | accuracy (%): 80.37 -> 81.52 | Improved |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Resnet18 student · Stanford Cars | accuracy (%): 90.74 -> 91.53 | Improved |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Mobilenetv2 student · CUB-200-2011 | accuracy (%): 81.48 -> 82.40 | Improved |
| replace unified temperature with feature-norm factor (KD-fn) | KD, Mobilenetv2 student · Stanford Cars | accuracy (%): 90.49 -> 91.10 | Improved |
| use a larger teacher (Resnet152 instead of Resnet50) with KD | KD, Mobilenetv2 student · CUB-200-2011 | accuracy (%): 81.48 -> 81.57 | No clear effect |
| use a larger teacher (Resnet152 instead of Resnet50) with KD-fn | KD-fn, Mobilenetv2 student · CUB-200-2011 | accuracy (%): 82.40 -> 82.80 | Improved |
| use feature-norm factor in self-distillation | self KD, Resnet20 · Cifar-100 | accuracy (%): 79.48 -> 80.16 | Improved |
| use feature-norm factor in self-distillation | self KD, Mobilenetv2 · CUB-200-2011 | accuracy (%): 80.32 -> 80.47 | Improved |
| add teacher distillation to hypersphere embedding (KD-fn instead of HE) | HE, Resnet20 · Cifar-100 | accuracy (%): 78.96 -> 81.19 | Improved |
Closest papers by meaning
- Improving Knowledge Distillation via Regularizing Feature Direction and Norm (ECCV 2024 · similarity 0.92)
- From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels (ICCV 2023 · similarity 0.91)
- Revisiting Knowledge Distillation via Label Smoothing Regularization (CVPR 2020 · similarity 0.90)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.90)
- Decoupled Knowledge Distillation (CVPR 2022 · similarity 0.90)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
