Research map / Label-efficient and robust learning / Knowledge distillation
Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification
Quanshi Zhang, Cheng Xu, Yilan Chen, Zhefan Rao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train by feature distillation instead of from scratch | VGG-16 FC1 · CUB200-2011 | N_point^fg: 22.50 -> 43.77 | Improved |
| train by feature distillation instead of from scratch | VGG-16 FC1 · CUB200-2011 | foreground ratio lambda: 0.68 -> 0.84 | Improved |
| train by feature distillation instead of from scratch | VGG-16 FC1 · CUB200-2011 | rho: 0.50 -> 0.66 | Improved |
| train by feature distillation instead of from scratch | VGG-16 FC1 · CUB200-2011 | D_var: 4.98 -> 0.06 | Improved |
| train by feature distillation instead of from scratch | BERT · SQuAD | N_point^fg: 3.58 -> 8.46 | Improved |
| train by feature distillation instead of from scratch | BERT · SST-2 | rho: 0.51 -> 0.80 | Improved |
| train by feature distillation instead of from scratch | PointNet++ FC1 · ModelNet40 with background points | N_point^fg: 16.04 -> 51.15 | Improved |
| train by feature distillation instead of from scratch | DGCNN FC1 · ModelNet40 with background points | N_point^fg: 102.53 -> 84.44 | Hurt |
| train by feature distillation instead of from scratch | VGG-16 FC1 · ILSVRC-2013 DET | N_point^fg: 36.47 -> 35.98 | No clear effect |
| distill from the output layer FC3 instead of scratch | VGG-16 FC3 · CUB200-2011 | lambda: 0.71 -> 0.63 | Hurt |
| use 8x8 grid instead of 16x16 for knowledge-point measurement (ablation) | VGG-16 FC1 student · CUB200-2011 | N_point^fg: student still above baseline 10.06 vs 6.47 | No clear effect |
| lower threshold b to 0.15 for knowledge-point measurement (ablation) | VGG-16 FC1 student · CUB200-2011 | N_point^fg: student still above baseline 54.53 vs 26.43 | No clear effect |
Closest papers by meaning
- Explaining Knowledge Distillation by Quantifying the Knowledge (CVPR 2020 · similarity 0.92)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.89)
- Harmonizing knowledge Transfer in Neural Network with Unified Distillation (ECCV 2024 · similarity 0.89)
- Knowledge Distillation: A Survey (IJCV 2021 · similarity 0.88)
- Class Attention Transfer Based Knowledge Distillation (CVPR 2023 · similarity 0.88)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
