Research map / Label-efficient and robust learning / Knowledge distillation
Class Attention Transfer Based Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer class activation maps with CAT-KD instead of attention transfer (AT) | Student CNNs on CIFAR-100 · CIFAR-100, five different-architecture teacher/student pairs | accuracy: +6.53, +4.03, +5.68, +12.78, +9.73 | Helped |
| Use CAT-KD rather than the strongest listed baseline | ResNet20 student on CIFAR-100 · CIFAR-100, ResNet56 → ResNet20 | accuracy: 71.97 -> 71.62 | Hurt |
| Use CAT-KD rather than the strongest listed baseline | ResNet8×4 student on CIFAR-100 · CIFAR-100, ResNet32×4 → ResNet8×4 | accuracy: ReviewKD 75.63 -> CAT-KD 76.91 | Helped |
| Use CAT-KD rather than the strongest listed baseline | ResNet18 student on ImageNet · ImageNet, ResNet34 → ResNet18 | Top-1 accuracy: 71.7 -> 71.26 | Hurt |
| Use CAT-KD rather than the strongest listed baseline | MobileNet student on ImageNet · ImageNet, ResNet50 → MobileNet | Top-5 accuracy: 91.05 -> 91.13 | Helped |
| Transfer CAMs produced by a more accurate teacher | ShuffleNetV1 trained with CAT · CIFAR-100 | accuracy: 72.47 -> 74.42 -> 76.17 | Helped |
| Pool transferred CAMs to 4×4 rather than 8×8 (ablation) | ResNet32×4 trained with CAT · CIFAR-100 | accuracy: 79.65 -> 79.84 | Helped |
| Pool transferred CAMs to 2×2 rather than 8×8 (ablation) | ResNet8×4 trained with CAT · CIFAR-100 | accuracy: 67.92 -> 72.45 | Helped |
| Replace real-valued CAMs with binarized CAMs (ablation) | ResNet50 trained with CAT · CIFAR-100 | accuracy: 80.45 -> 79.65 | Hurt |
| Remove normalization of transferred CAMs (ablation) | ShuffleNetV1 student with a different-architecture teacher · CIFAR-100, ResNet32×4 → ShuffleNetV1 | accuracy: 77.78 -> 78.26 | Helped |
| Normalize transferred CAMs (ablation) | students with similar architecture to their teachers · CIFAR-100, three teacher/student pairs | accuracy: 73.45 -> 73.62; 75.46 -> 75.6; 76.29 -> 76.91 | Helped |
| Use CAT loss without cross-entropy when the CAM producer is weaker than the trained model (ablation) | Models trained with CAT · CIFAR-100 | accuracy | Hurt |
Closest papers by meaning
- ALP-KD: Attention-Based Layer Projection for Knowledge Distillation (AAAI 2021 · similarity 0.90)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.89)
- Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model (ICML 2021 · similarity 0.89)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.87)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.