Research map / Detection and segmentation / CNN features
DetKDS: Knowledge Distillation Search for Object Detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DetKDS searches over transforms, distance functions and loss weights for detection knowledge distillation using a divide-and-conquer evolutionary algorithm, yielding distillers that transfer across homogeneous and heterogeneous detector pairs on COCO.
Problem
Designing effective knowledge distillation recipes for object detectors is hard because teacher-student pairs differ in backbone, paradigm and label assignment, and manual tuning of distillers and loss weights is costly.
Approach
Treats detection distiller design as a search problem over a large space of feature/response inputs, cascaded transforms, distance functions and loss weights, and searches it with a divide-and-conquer evolutionary algorithm (first per-loss distiller, then combined weights) using proxy evaluation on COCO subsets.
Key results
- On COCO, DetKDS raises Faster R-CNN-R50 from 38.4 to 42.5 AP (+4.1) with ResNeXt-101 teacher, and RetinaNet-R50 from 37.4 to 41.1 (+3.7).
- DetKDS improves FCOS-R50 by +4.0 AP (38.5 to 42.5) with ResNet-101 teacher, and RepPoints-R50 by +3.7 AP.
- On Deformable DETR-R50, DetKDS reaches 46.2 AP vs 44.1 baseline (+2.1), outperforming FGD and MGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global feature distillation loss (ablation) | RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 37.4 -> 41.0 | Helped |
| Add foreground-background feature distillation loss (ablation) | RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 37.4 -> 40.7 | Helped |
| Add instance feature distillation loss (ablation) | RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 37.4 -> 40.9 | Helped |
| Add localization response distillation loss (ablation) | RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 37.4 -> 39.4 | Helped |
| Add logits response distillation loss (ablation) | RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 37.4 -> 38.6 | Helped |
| Remove divide-and-conquer from evolutionary search (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 41.1 -> 40.4 | Hurt |
| Remove the distiller filter (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 41.1 -> 39.8 | Hurt |
| Replace evolutionary search with Monte Carlo tree search (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 41.1 -> 40.8 | Hurt |
| Replace evolutionary search with random search (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 41.1 -> 40.5 | Hurt |
| Increase search iterations from 20 to 30 (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 40.8 -> 40.9 | Helped |
| Increase search iterations from 30 to 40 (ablation) | DetKDS for RetinaNet-R50 distilled from RetinaNet-X101 · COCO val | AP: 40.9 -> 41.1 | Helped |
| Apply DetKDS distillation (ablation) | RetinaNet-R18 with RetinaNet-X101 teacher · COCO val | AP: 32.6 -> 36.1 | Helped |
Surprising findings
- Response (logits and localization) distillation is weaker than feature distillation on detection; global feature distillation alone gives the largest single-loss gain.
- Simple distance functions such as l1 generalize across detectors better than more complex options.
Limitations
- Search adds extra computational cost compared with manual distiller design.
- Search space of over 30,000 candidates is fixed and hand-designed, so discovered distillers are bounded by its options.
Experimental setup
Datasets: COCO
Benchmarks: COCO val2017
Metrics: AP AP50 AP75 APS APM APL Mask AP
Baselines: FitNets GID FRS FGD PKD Auto-KD DiffKD MasKD Defeat LD SKD MGD
Builds on: Auto-KD FGD PKD GID LD
Closest papers by meaning
- Joint-DetNAS: Upgrade Your Detector With NAS, Pruning and Dynamic Distillation (CVPR 2021 · similarity 0.91)
- Distilling Image Classifiers in Object Detectors (NeurIPS 2021 · similarity 0.91)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.89)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.88)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.88)
- CrossKD: Cross-Head Knowledge Distillation for Object Detection (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.