Research map / Label-efficient and robust learning / Knowledge distillation
A Fast Knowledge Distillation Framework for Visual Recognition
Zhiqiang Shen, Eric Xing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FKD soft labels instead of ReLabel label maps | ResNet-50 from scratch · ImageNet-1K | Top-1: 78.9 -> 79.8 | Improved |
| use FKD soft labels instead of ReLabel label maps | ResNet-101 from scratch · ImageNet-1K | Top-1: 80.7 -> 81.7 | Improved |
| train MEAL V2 with FKD instead of online teacher forwards | MEAL V2 ResNet-50 · ImageNet-1K | Top-1: 80.67 -> 80.70 | No clear effect |
| use cosine lr and 1.5x epochs with FKD (ablation) | MEAL V2 w/ FKD ResNet-50 · ImageNet-1K | Top-1: 80.70 -> 80.91 | Improved |
| apply FKD to ViT-T without extra augmentation instead of DeiT KD | ViT-T · ImageNet-1K | Top-1: 74.5 -> 75.2 | Improved |
| apply FKD to SReT-LT instead of its original KD | SReT-LT · ImageNet-1K | Top-1: 77.7 -> 78.7 | Improved |
| use FKD instead of S2-BNN for self-supervised distillation | ReActNet · ImageNet-1K linear eval | Top-1: 61.5 -> 61.7 | No clear effect |
| use FKD instead of S2-BNN for self-supervised distillation | ResNet-50 · ImageNet-1K linear eval | Top-1: 68.7 -> 68.8 | No clear effect |
| sample 4 crops per image in a mini-batch instead of 1 (ablation) | vanilla ResNet-50 · ImageNet-1K | Top-1: 77.18 -> 78.14 | Improved |
| sample 32 crops per image in a mini-batch instead of 1 (ablation) | FKD ResNet-50 from scratch · ImageNet-1K | Top-1: 79.59 -> 74.61 | Hurt |
| use full soft labels instead of hard labels (ablation) | MEAL V2 w/ FKD ResNet-50 · ImageNet-1K | Top-1: 80.20 -> 80.65 | Improved |
| use Marginal Smoothing (K=5) instead of Marginal Smoothing (K=10) (ablation) | FKD from scratch ResNet-50 · ImageNet-1K | Top-1: 79.44 -> 79.51 | No clear effect |
Closest papers by meaning
- From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels (ICCV 2023 · similarity 0.91)
- Multi-label Self Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Feature Normalized Knowledge Distillation for Image Classification (ECCV 2020 · similarity 0.89)
- Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning (CVPR 2023 · similarity 0.89)
- Knowledge Distillation As Efficient Pre-Training: Faster Convergence, Higher Data-Efficiency, and Better Transferability (CVPR 2022 · similarity 0.89)
- Self Supervision to Distillation for Long-Tailed Visual Recognition (ICCV 2021 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
