Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation via Instance Relationship Graph
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, Yunqiang Duan
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill instance features and instance relationships using IRG (ablation) | ResNet20-x0.5 · CIFAR10 | accuracy: 88.36 -> 90.28 | Improved |
| Add IRG transformation distillation to LIRG (ablation) | ResNet20-x0.5 · CIFAR10 | accuracy: 90.28 -> 90.69 | Improved |
| Use one-to-many instead of one-to-one layer supervision (ablation) | LIRG with ResNet20-x0.5 student and ResNet20 teacher · CIFAR10 | accuracy: 90.03 -> 90.28 | Improved |
| Use one-to-many instead of one-to-one layer supervision (ablation) | LIRG with ResNet20-x0.5 student and ShuffleNet-x0.5 teacher · CIFAR10 | accuracy: 89.89 -> 90.21 | Improved |
| Use one-to-many supervision with three student layers rather than one (ablation) | LIRG with ResNet20-x0.5 student and ResNet20 teacher · CIFAR10 | accuracy: 89.87 -> 90.28 | Improved |
| Use one-to-many supervision with five student layers rather than three (ablation) | LIRG with ResNet20-x0.5 student and ResNet20 teacher · CIFAR10 | accuracy: 90.28 -> 90.02 | Hurt |
| Add IRG distillation rather than training only with ground-truth loss (ablation) | ResNet20-x0.5 · CIFAR100-coarse | accuracy: 72.51 -> 74.32 | Improved |
| Add IRG transformation distillation to LIRG (ablation) | ResNet20-x0.5 · CIFAR100-coarse | accuracy: 74.32 -> 74.64 | Improved |
| Add IRG distillation rather than training only with ground-truth loss (ablation) | ResNet20-x0.5 · CIFAR100-fine | accuracy: 59.88 -> 61.93 | Improved |
| Add IRG transformation distillation to LIRG (ablation) | ResNet20-x0.5 · CIFAR100-fine | accuracy: 61.93 -> 62.25 | Improved |
| Use LMTK rather than Rocket distillation | ResNet14-x0.5 · CIFAR10-small | accuracy: 62.23 -> 66.04 | Improved |
| Use LMTK rather than Rocket distillation | ResNet20-x0.375 · ImageNet | accuracy: 76.16 -> 77.18 | Improved |
Closest papers by meaning
- Distilling Holistic Knowledge With Graph Neural Networks (ICCV 2021 · similarity 0.90)
- Distilling Knowledge via Knowledge Review (CVPR 2021 · similarity 0.88)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.88)
- Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (AAAI 2021 · similarity 0.87)
- Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion (AAAI 2024 · similarity 0.86)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
