Research map / Label-efficient and robust learning / Knowledge distillation
DETRDistill: A Universal Knowledge Distillation Framework for DETR-families
Jiahao Chang, Shuo Wang, Hai-Ming Xu, Zehui Chen, Chenhongyi Yang, Feng Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add negative-prediction box distillation to logits KD (ablation) | DETRDistill logits distillation · COCO val2017 AdaMixer | AP: 42.6 -> 43.6 (pos reg vs pos+neg reg) | Improved |
| distill only positive predictions (ablation) | logits KD · COCO val2017 AdaMixer | AP: 42.3 -> 42.6 | No clear effect |
| distill stage-by-stage instead of teacher's last stage only (ablation) | logits KD · COCO val2017 AdaMixer | AP: 43.1 -> 43.7 | Improved |
| weight teacher queries by quality score in feature KD mask (ablation) | target-aware feature distillation · COCO val2017 AdaMixer | AP: 41.9 -> 43.5 | Improved |
| add query-prior assignment distillation to LD + FD (ablation) | DETRDistill · COCO val2017 AdaMixer | AP: 44.3 -> 44.7 | Improved |
| apply CNN feature distillation FGD to DETR student | AdaMixer · COCO val2017 12 epochs | AP: 42.3 -> 40.7 | Hurt |
| apply DETRDistill | Deformable DETR R50 · COCO val2017 50 epochs | AP: 44.1 -> 46.6 | Improved |
| apply DETRDistill | Conditional DETR R50 · COCO val2017 50 epochs | AP: 40.7 -> 42.9 | Improved |
| reduce decoder stages from 6 to 3 in student (ablation) | Conditional DETR student · COCO val2017 12 epochs | AP: 33.0 -> 30.3 | Hurt |
| reduce encoder stages from 6 to 3 in student (ablation) | Conditional DETR student · COCO val2017 12 epochs | AP: 33.0 -> 33.0 | No clear effect |
| apply DETRDistill to 3/3 stage student | Conditional DETR student · COCO val2017 12 epochs | AP: 29.9 -> 35.7 | Improved |
Later papers that compared with this method
2 later accepted papers report DETRDistill as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Distilling Knowledge from Large-Scale Image Models for Object Detection (ECCV 2024 · better on 2 of 2 shared results)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.92)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.92)
- Distilling Knowledge from Large-Scale Image Models for Object Detection (ECCV 2024 · similarity 0.92)
- Distilling Object Detectors via Monte Carlo Dropout (TPAMI 2026 · similarity 0.89)
- End-to-End Object Detection with Transformers (ECCV 2020 · similarity 0.89)
- MS-DETR: Efficient DETR Training with Mixed Supervision (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
