Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Knowledge from Large-Scale Image Models for Object Detection
Gang Li, Wenhai Wang, Xiang Li, Ziheng Li, Jian Yang, Jifeng Dai, Yu Qiao, Shanshan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a frozen-backbone teacher instead of a fully fine-tuned teacher with Query Distillation (ablation) | DLIM-Det (Swin-T student) · COCO val2017 | mAP: 51.6 -> 52.2 | Improved |
| inherit teacher detection head parameters (ablation) | DLIM-Det (Swin-T student) · COCO val2017 | mAP: 54.4 vs 52.2 without head inheriting | Improved |
| freeze teacher backbone while matching teacher mAP (56.7 vs 56.9) (ablation) | Swin-L teacher · COCO val2017 | student mAP: 53.1 -> 54.4 | Improved |
| freeze backbone in teacher training | Swin-L teacher detection performance · COCO val2017 | teacher mAP: 58.1 -> 56.6 | Hurt |
| add Query Position Distill and Query Relation Distill together (ablation) | Swin-T student with Swin-B teacher · COCO val2017 | mAP: 51.3 -> 52.1 | Improved |
| apply Query Distillation on ResNet-50 student | DINO ResNet-50 · COCO val2017 | mAP: 48.9 -> 49.8 | Improved |
| distill on all 6 decoder layers instead of fewer layers (ablation) | Query Distillation · COCO val2017, Intern-T student | mAP: 54.1 -> 54.5 | Improved |
| distill from randomly sampled 300 teacher queries instead of all 900 (ablation) | Query Distillation · COCO val2017, Intern-T student | mAP: 54.2 -> 54.5 | Improved |
| distill from foreground-only or background-only teacher queries instead of random sample (ablation) | Query Distillation · COCO val2017, Intern-T student | mAP: 54.5 -> 54.2 | Hurt |
| use larger teacher (Swin-L frozen instead of Swin-B frozen) (ablation) | DLIM-Det · COCO val2017, Swin-T student | mAP: 53.5 -> 54.4 | Improved |
| apply prior KD (FGD) with InternImage large teacher | InternImage-T student · COCO val2017 | mAP: 53.2 -> 52.8 | Hurt |
| distill directly from large teacher instead of two-step teacher assistant | Swin-T student · COCO val2017 | mAP: 53.7 -> 54.4 | Improved |
Closest papers by meaning
- DETRDistill: A Universal Knowledge Distillation Framework for DETR-families (ICCV 2023 · similarity 0.92)
- Structured Knowledge Distillation for Accurate and Efficient Object Detection (TPAMI 2023 · similarity 0.91)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.90)
- Prediction-Guided Distillation for Dense Object Detection (ECCV 2022 · similarity 0.90)
- Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models (NeurIPS 2024 · similarity 0.88)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
