Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Knowledge via Knowledge Review
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace same-stage-only distillation with cross-stage knowledge review (ablation) | ReviewKD · CIFAR-100 (WRN40-2→WRN16-2) | accuracy: 74.3 (5e-2) -> 75.2 (6e-2) | Helped |
| Add the residual learning framework to knowledge review (ablation) | ReviewKD · CIFAR-100 (WRN40-2→WRN16-2) | accuracy: 75.2 (6e-2) -> 75.6 (6e-2) | Helped |
| Add attention-based fusion (ABF) (ablation) | ReviewKD with residual learning · CIFAR-100 (WRN40-2→WRN16-2) | accuracy: 75.6 (6e-2) -> 76.0 (6e-2) | Helped |
| Add hierarchical context loss (HCL) (ablation) | ReviewKD with residual learning · CIFAR-100 (WRN40-2→WRN16-2) | accuracy: 75.6 (6e-2) -> 75.8 (5e-2) | Helped |
| Combine ABF and HCL (ablation) | ReviewKD with residual learning · CIFAR-100 (WRN40-2→WRN16-2) | accuracy: 76.0 (6e-2) -> 76.2 (4e-2) | Helped |
| Use lower teacher stages to supervise deeper student stages (ablation) | ResNet20 student with ResNet56 teacher · CIFAR-100 | accuracy: 69.1 -> 69.2 | Helped |
| Use higher teacher stages to supervise earlier student stages (ablation) | ResNet20 student with ResNet56 teacher · CIFAR-100 | accuracy: 69.1 -> 68.2 | Hurt |
| Use knowledge review distillation instead of the strongest listed baseline | ReviewKD · CIFAR-100 (same-style architectures) | accuracy: 71.16 -> 71.89 | Helped |
| Use knowledge review distillation instead of the strongest listed baseline | ReviewKD · ImageNet (ResNet50→MobileNet) | Top-1: 71.37 -> 72.56 | Helped |
| Use knowledge review distillation instead of the strongest listed baseline | ReviewKD · COCO2017 val (Faster R-CNN R101-FPN→R18-FPN) | mAP: 35.44 (+2.18) -> 36.75 (+3.49) | Helped |
| Use knowledge review distillation | Mask R-CNN R101-FPN→R18-FPN · COCO2017 val instance segmentation | mAP: 31.25 -> 33.62 (+2.37) | Helped |
Closest papers by meaning
- Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (AAAI 2021 · similarity 0.91)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.88)
- CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.