Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge distillation for detectors
Distill detectors by mimicking features, rankings, and focal or global knowledge.
25 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Knowledge distillation, keyword cluster “teacher student · kd · teachers”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
5 papers in 2023–24 and 3 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.5% to 1.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Learning Efficient Object Detection Models with Knowledge Distillation (NeurIPS 2017)
- 2019 · Distilling Object Detectors With Fine-Grained Feature Imitation (CVPR 2019 · 466 citations)
- 2020 · Online Knowledge Distillation via Collaborative Learning (CVPR 2020 · 329 citations)
- 2021 · Distilling Knowledge via Knowledge Review (CVPR 2021 · 568 citations)
- 2022 · Focal and Global Knowledge Distillation for Detectors (CVPR 2022 · 377 citations)
- 2023 · Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation (ICML 2023 · 3 citations)
- 2024 · CrossKD: Cross-Head Knowledge Distillation for Object Detection (CVPR 2024 · 132 citations)
- 2025 · DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · 10 citations)
- 2026 · Distilling Future Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection (AAAI 2026 · 0 citations)
Earliest papers here
- Learning Efficient Object Detection Models with Knowledge Distillation (NeurIPS 2017)
- Distilling Object Detectors With Fine-Grained Feature Imitation (CVPR 2019 · 466 citations)
- Online Knowledge Distillation via Collaborative Learning (CVPR 2020 · 329 citations)
What has been tried
22 papers here report 293 measured changes (a component added, removed or swapped, compared with a baseline): helped 80%, no clear effect 4%, hurt 14%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Apply CrossKD instead of direct prediction mimicking | COCO minival · AP | 37.8 -> 38.7 | Helped | CrossKD: Cross-Head Knowledge Distillation for Object Detection (CVPR 2024 · 132 citations) |
| Use inconsistent object-query permutations for distillation | MS COCO2017 · AP | 36.2 -> 35.1 | Hurt | KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · 26 citations) |
| Use an orthogonal projection instead of the standard distillation setup | ImageNet-1K · Top-1 accuracy | 76.3 -> 77.9 | Helped | VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · 17 citations) |
| Add adaptive channel attention weights to CKD | MS COCO · mAP | 40.6 -> 41.0 | Helped | DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · 10 citations) |
| distill only high-disparity regions instead of the entire feature map | COCO · mAP | 36.4 -> 39.8 | Helped | DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 5 papers of this approach from 2024–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MS COCO2017 · 2 of 5 papers
Baselines
- FGD · 4 of 5 papers
- PKD · 4 of 5 papers
- GID · 3 of 5 papers
- CWD · 2 of 5 papers
- FitNets · 2 of 5 papers
- Defeat · 2 of 5 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 8 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 2 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Rethinking distillation objectives (160 papers)
- Distillation with causal and adaptive guidance (82 papers)
- Few-shot and metric distillation (80 papers)
- Teacher properties and robustness (54 papers)
Elsewhere in the topic
- Patient consistent distillation (68 papers)
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).