Research map / Label-efficient and robust learning / Knowledge distillation
Curriculum Temperature for Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic adversarial curriculum temperature to vanilla KD | KD · CIFAR-100, ResNet-56 → ResNet-20 | Top-1 accuracy: 70.66 -> 71.19 | Helped |
| Add CTKD to existing distillation methods | PKT, SP, VID, CRD, SRRL, and DKD · CIFAR-100 | Top-1 accuracy: +0.31; +0.43; +0.13; +0.42; +0.32; +0.22 | Helped |
| Add CTKD to KD | KD · ImageNet-2012, ResNet-34 → ResNet-18 | Top-1 accuracy: 70.83 -> 71.32 | Helped |
| Add CTKD to KD | Faster R-CNN-FPN KD · MS-COCO val2017, ResNet-101 → ResNet-18 | mAP: 33.97 -> 34.56 | Helped |
| Use instance-wise rather than global learned temperature (ablation) | CTKD · CIFAR-100, ResNet-56 → ResNet-20 | Top-1 accuracy: 71.19 -> 71.32 | Helped |
| Replace the dynamic curriculum with fixed λ = 5 (ablation) | CTKD · CIFAR-100, ResNet-110 → ResNet-32 | Top-1 accuracy: 73.52 -> 72.78 | Hurt |
| Begin with λ = 1 instead of λ = 0 in the curriculum (ablation) | CTKD · CIFAR-100, ResNet-110 → ResNet-32 | Top-1 accuracy: 73.52 -> 72.58 | Hurt |
| Use cosine rather than linear increase of λ over 10 epochs (ablation) | CTKD · CIFAR-100, ResNet-110 → ResNet-32 | Top-1 accuracy: 73.31 -> 73.52 | Helped |
| Use cosine curriculum rather than vanilla distillation during the first 10 epochs (ablation) | CTKD · CIFAR-100, ResNet-110 → ResNet-32 | Top-1 accuracy: 73.21 -> 73.52 | Helped |
| Add adversarial temperature learning while holding λ fixed at 1 (ablation) | Vanilla KD · CIFAR-100, ResNet-56 → ResNet-20 | Top-1 accuracy: 70.66 -> 71.01 | Helped |
| Add curriculum distillation to adversarial temperature learning (ablation) | Adversarial temperature-only KD · CIFAR-100, ResNet-56 → ResNet-20 | Top-1 accuracy: 71.01 -> 71.19 | Helped |
| Increase instance-temperature MLP intermediate channels from 64 to 256 (ablation) | Instance-T · CIFAR-100, ResNet-56 → ResNet-20 | Top-1 accuracy: 71.28 -> 71.32 | Helped |
Closest papers by meaning
- Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.87)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.