Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Holistic Knowledge With Graph Neural Networks
Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, Jiajun Bu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HKD holistic distillation on top of KD | KD, ResNet32x4 -> ResNet8x4 · CIFAR100 | top-1 accuracy (%): 73.55 -> 76.13 | Improved |
| add HKD holistic distillation on top of KD | KD, ResNet50 -> MobileNetV2 · CIFAR100 | top-1 accuracy (%): 67.83 -> 70.72 | Improved |
| add the KD loss to HKD | HKD, ResNet32x4 -> ResNet8x4 · CIFAR100 | top-1 accuracy (%): 75.63 -> 76.13 | Improved |
| add the KD loss to HKD | HKD, ResNet50 -> VGG8 · CIFAR100 | top-1 accuracy (%): 74.86 -> 74.88 | No clear effect |
| replace CRD+KD with HKD+KD | student network, ResNet32x4 -> ShuffleNetV2 · TinyImageNet | top-1 accuracy (%): 58.95 -> 59.31 | Improved |
| replace CRD with HKD | student network, ResNet34 teacher to ResNet18 student · ImageNet | top-1 accuracy (%): 53.76 -> 54.07 | Improved |
| replace CRD+KD with HKD+KD | MobileNetV2 student transferred from CIFAR100 · TinyImageNet linear probe | accuracy (%): 28.98 -> 30.55 | Improved |
| replace CRD+KD with HKD+KD | MobileNetV2 student transferred from CIFAR100 · STL-10 linear probe | accuracy (%): 65.87 -> 67.28 | Improved |
| add vanilla KD to the student | MobileNetV2 student transferred from CIFAR100 · TinyImageNet linear probe | accuracy (%): 23.01 -> 22.92 | No clear effect |
| use a two-layer GNN (L=2) instead of one layer (ablation) | HKD+KD, ResNet32x4 -> ResNet8x4 · CIFAR100 | top-1 accuracy (%): 76.13 -> 76.05 | No clear effect |
| use a two-layer GNN (L=2) instead of one layer (ablation) | HKD+KD, VGG13 -> MobileNetV2 · CIFAR100 | top-1 accuracy (%): 70.48 -> 70.28 | No clear effect |
| use a two-layer GNN (L=2) instead of one layer (ablation) | HKD+KD, ResNet50 -> VGG8 · CIFAR100 | top-1 accuracy (%): 74.85 -> 74.82 | No clear effect |
Closest papers by meaning
- Multi-Scale Distillation from Multiple Graph Neural Networks (AAAI 2022 · similarity 0.91)
- Pareto-Based Heterogeneous Knowledge Distillation for MLPs on Graphs (AAAI 2026 · similarity 0.90)
- Harmonizing knowledge Transfer in Neural Network with Unified Distillation (ECCV 2024 · similarity 0.90)
- Knowledge Distillation via Instance Relationship Graph (CVPR 2019 · similarity 0.90)
- Heterogeneous Knowledge Distillation Using Information Flow Modeling (CVPR 2020 · similarity 0.89)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
