Research map / Label-efficient and robust learning / Knowledge distillation
KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KD-Zero searched distiller instead of handcrafted KD | ResNet-20 (R110 teacher) · CIFAR-100 | top-1 accuracy: 70.67 -> 72.05 | Helped |
| use KD-Zero in cross-architecture pairs (ViT students, CNN teacher) | DeiT-Ti · CIFAR-100 | top-1 accuracy: 73.25 -> 78.25 | Helped |
| use KD-Zero instead of KD on ImageNet | ResNet-18 · ImageNet | top-1 accuracy: 70.66 -> 72.17 | Helped |
| use KD-Zero on detection students | RetinaNet-R50 · COCO val | AP: 37.4 -> 40.9 | Helped |
| use KD-Zero distillation on segmentation | PSPNet-R18 · Cityscapes val | mIoU: 72.55 -> 76.25 | Helped |
| use KD-Zero loss in place of default loss of Review (ablation) | Review on ResNet-20 · CIFAR-100 | top-1 accuracy: 71.89 -> 72.27 | Helped |
| use KD-Zero loss with BYOT self-distillation (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy: 70.37 -> 70.98 | Helped |
| reuse the searched distiller with stronger R32x4 teacher (ablation) | KD-Zero on ResNet-20 · CIFAR-100 | top-1 accuracy: 72.05 (R110) -> 70.98 (R32x4) | Mixed |
| evolutionary search instead of random search (ablation) | KD-Zero distiller search · CIFAR-100 ResNet-20 | search quality | Helped |
| remove transform, weight factor or temperature factor search (ablation) | KD-Zero · CIFAR-100 ResNet-20 | top-1 accuracy | Hurt |
| add sharpness and CKA gap objectives to cross-entropy in fitness (ablation) | KD-Zero search · CIFAR-100 | evaluation quality | Helped |
Closest papers by meaning
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.89)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.