Research map / Label-efficient and robust learning / Knowledge distillation
Prime-Aware Adaptive Distillation
Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang, Yichen Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hard-mining weighting (large weights for hard samples) (ablation) | L2 distillation · CIFAR100 | top-1 accuracy (%): 72.85 -> 72.12 | Hurt |
| add hard-mining weighting (ablation) | L2 distillation · CUB-200 | R@1 (%): 58.6 -> 55.7 | Hurt |
| add hard-mining weighting (ablation) | L2 distillation · TinyImageNet | top-1 accuracy (%): 63.08 -> 62.57 | Hurt |
| discard the hardest samples in each batch (ablation) | L2 distillation · CIFAR100 | top-1 accuracy (%): 72.85 -> 72.91 | No clear effect |
| discard the hardest samples in each batch (ablation) | L2 distillation · Pascal VOC07 | mmAP: 45.53 -> 45.94 | Improved |
| add soft-exp weighting (ablation) | L2 distillation · CIFAR100 | top-1 accuracy (%): 72.85 -> 73.54 | Improved |
| add soft-poly weighting (ablation) | L2 distillation · CUB-200 | R@1 (%): 58.6 -> 60.4 | Improved |
| vary hyper-parameter of soft-exp weighting (0.1 to 10) (ablation) | L2 distillation · TinyImageNet | top-1 accuracy (%): 63.1 baseline; 61.2 at 0.1, 63.6 at 5 | Mixed |
| use PAD learned weights instead of soft-exp (ablation) | L2 distillation · TinyImageNet | top-1 accuracy (%): 63.64 -> 65.54 | Improved |
| add full PAD (variance + log term) over weights from PAD (ablation) | L2 distillation · Pascal VOC07 | mmAP: 46.52 -> 46.92 | Improved |
| add warm up to L2 distillation (ablation) | L2 distillation · CIFAR100 | top-1 accuracy (%): 72.85 -> 73.13 | Improved |
| add PAD to L2 distillation | L2 distillation · CIFAR100 | top-1 accuracy (%): 72.85 -> 74.06 | Improved |
Closest papers by meaning
- EA-KD: Entropy-based Adaptive Knowledge Distillation (ICCV 2025 · similarity 0.88)
- What to Distill? Fast Knowledge Distillation with Adaptive Sampling (ICCV 2025 · similarity 0.88)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.87)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.86)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.86)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
