Research map / Label-efficient and robust learning / Knowledge distillation
Local Correlation Consistency for Knowledge Distillation
Xiaojie Li, Jianlong Wu, Hongyu Fang, Yue Liao, Fei Wang, Chen Qian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intra-patch local correlation loss to KD (ablation) | KD, ResNet14/ResNet20 students · CIFAR100 | top-1 accuracy: 69.78 -> 70.00 (ResNet14); 71.47 -> 72.04 (ResNet20) | Improved |
| add inter-same local correlation loss to KD (ablation) | KD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 69.78 -> 70.20; 71.47 -> 71.96 | Improved |
| add inter-diff local correlation loss to KD (ablation) | KD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 69.78 -> 70.03; 71.47 -> 72.10 | Improved |
| combine all three local losses (ablation) | KD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 69.78 -> 70.37; 71.47 -> 72.31 | Improved |
| add class-aware attention module (CAAT) (ablation) | local-loss KD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 70.37 -> 70.48; 72.31 -> 72.63 | Improved |
| increase patch grid k from 1 to 4 (ablation) | local correlation loss, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 69.98 -> 70.37; 71.82 -> 72.31 | Improved |
| use CAAT instead of AT attention (ablation) | LKD, ResNet14 · CIFAR100 | top-1 accuracy: 69.88 -> 70.48 | Improved |
| use CAAT instead of Grad-CAM attention (ablation) | LKD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 70.41 -> 70.48; 72.18 -> 72.63 | Improved |
| use AT activation attention instead of no attention (ablation) | LKD, ResNet14/ResNet20 · CIFAR100 | top-1 accuracy: 70.37 -> 69.88; 72.31 -> 72.34 | Mixed |
| replace global correlation (SP, CC) with local correlation (LKD) | KD students · CIFAR100 | top-1 accuracy: 69.77 -> 70.48 (ResNet14, CC vs LKD) | Improved |
| replace KD with LKD | ResNet18 student · ImageNet | top-1 accuracy: 71.34 -> 71.54 | Improved |
| replace CC with LKD | ResNet18 student · ImageNet | top-5 accuracy: 90.26 -> 90.30 | Improved |
Closest papers by meaning
- Correlation Congruence for Knowledge Distillation (ICCV 2019 · similarity 0.90)
- Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (AAAI 2021 · similarity 0.90)
- Distilling Knowledge via Knowledge Review (CVPR 2021 · similarity 0.89)
- Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence (ICML 2025 · similarity 0.87)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.87)
- CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
