Research map / Label-efficient and robust learning / Knowledge distillation
KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KD-DETR, a logit-based knowledge distillation paradigm for DETR that uses shared, unlearnable specialized object queries as consistent distillation points, improving DAB/Deformable/DINO students on COCO by 2.6-5.2 AP.
Problem
Knowledge distillation works poorly on DETR-style detectors because object queries in teacher and student are egocentric and inconsistent in number and correspondence, so the distillation points (inputs whose predictions are mimicked) are insufficient and unreliable.
Approach
Decouple detection and distillation by introducing a set of shared, unlearnable distillation-point queries (general random-initialized queries plus teacher's own queries) fed to both teacher and student decoders, weighted by teacher foreground probability, so the student mimics consistent teacher predictions; extended to DETR-to-CNN heterogeneous distillation via IoU-selected anchors.
Key results
- DAB-DETR ResNet-18 student reaches 41.4 AP (+5.2 over 36.2 baseline); ResNet-50 student reaches 45.7 AP.
- Deformable DETR ResNet-18 student reaches 43.7 AP (+3.6) and ResNet-50 student 48.3 AP (+3.8), outperforming DETRDistill (46.6) by 1.7 AP on the ResNet-50 setting.
- DINO student gains +4.4 AP (ResNet-18) and +2.6 AP (ResNet-50); DINO-to-Faster R-CNN heterogeneous distillation gives +2.1 AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KD-DETR distillation from DAB-DETR ResNet-50 | DAB-DETR ResNet-18 · MS COCO2017 val | AP: 36.2 -> 41.4 | Helped |
| Use KD-DETR distillation from Deformable DETR ResNet-101 | Deformable DETR ResNet-50 · MS COCO2017 val | AP: 44.5 -> 48.3 | Helped |
| Distill from DINO ResNet-50 using KD-DETR | Faster R-CNN ResNet-50 · MS COCO2017 val | AP: 38.4 -> 40.5 | Helped |
| Use inconsistent object-query permutations for distillation (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 35.1 | Hurt |
| Use matched foreground queries as distillation points (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 37.2 | Helped |
| Add an average negative-query background distillation point (ablation) | DAB-DETR ResNet-18 with Similar Foreground points · MS COCO2017 | AP: 37.2 -> 37.4 | Helped |
| Increase object queries from 300 to 600 without distillation points (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 37.7 | Helped |
| Add 300 distillation points (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 40.2 | Helped |
| Use randomly initialized general distillation queries (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 38.7 | Helped |
| Use teacher queries as specific distillation points (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 39.9 | Helped |
| Add foreground rebalance weight to general sampling (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 38.7 -> 40.0 | Helped |
| Combine general sampling, specific sampling, and foreground rebalance weight (ablation) | DAB-DETR ResNet-18 · MS COCO2017 | AP: 36.2 -> 41.4 | Helped |
Surprising findings
- Students with ResNet-50 DAB-DETR and Deformable DETR slightly surpass their teachers (e.g., DAB-DETR student 45.7 vs teacher 42.1 AP in the table, text cites 2.2 AP gain).
- Simply increasing the number of object queries yields only trivial gains, so the improvement comes from distillation rather than extra queries.
Limitations
- Heterogeneous distillation gains are modest (+2.1 AP) and rely on a simple IoU-based anchor sampling heuristic.
- Dense general distillation points add background noise and degrade performance beyond about 300 points.
- Evaluation limited to COCO and ResNet backbones in the main experiments.
Experimental setup
Datasets: MS COCO2017 ImageNet1K (backbone pretraining)
Benchmarks: MS COCO2017
Metrics: AP AP50 AP75 APS APM APL
Baselines: DETRDistill FGD FitNet FGFI GID Baseline student without distillation
Builds on: DETR DAB-DETR Deformable DETR DINO logit-based distillation (Chen et al. 2017 style, cited as [4])
Closest papers by meaning
- Continual Detection Transformer for Incremental Object Detection (CVPR 2023 · similarity 0.89)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.89)
- Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective (CVPR 2024 · similarity 0.88)
- DETRs Beat YOLOs on Real-time Object Detection (CVPR 2024 · similarity 0.87)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.86)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.