Research map / Label-efficient and robust learning / Knowledge distillation
Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DPK feature distillation to ResNet18 using ResNet34 teacher | ResNet18 · ImageNet val | top-1 accuracy: 69.55 -> 72.51 | Helped |
| Add DPK feature distillation to MobileNetV2 using ResNet50 teacher | MobileNetV2 · ImageNet val | top-1 accuracy: 68.87 -> 73.26 | Helped |
| Add DPK to FGD for R50-FPN student with R101-FPN teacher | RetinaNet · COCO val | mAP: 39.6 -> 39.7 | Helped |
| Use dynamic mask ratio instead of fixed ratios (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.38 -> 72.46 | Helped |
| Use dynamic mask ratio instead of fixed ratios (ablation) | DPK, ResNet18 student/ResNet101 teacher · ImageNet | top-1 accuracy: 72.60 -> 72.87 | Helped |
| Replace random masking with block-wise or grid-wise masking (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.36 -> 72.36; 72.36 -> 72.32 | No clear effect |
| Use 1-CKA rather than 1-cosine similarity for dynamic masking (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.38 -> 72.46 | Helped |
| Use 1-CKA rather than exponential or linear decay schedules (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.35 -> 72.46; 72.36 -> 72.46 | Helped |
| Replace teacher-provided prior features with zero-padding (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.16 -> 72.46 | Hurt |
| Replace teacher-provided prior features with a learnable mask token (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.28 -> 72.46 | Hurt |
| Use ViT encoder-decoder rather than FitNets-style or convolutional transform (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 71.08 -> 72.46; 71.39 -> 72.46 | Helped |
| Add the encoder to MLP feature alignment (ablation) | DPK, ResNet18 student/ResNet34 teacher · ImageNet | top-1 accuracy: 72.31 -> 72.46 | Helped |
Closest papers by meaning
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.89)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.