Research map / Label-efficient and robust learning / Knowledge distillation
CrossKD: Cross-Head Knowledge Distillation for Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossKD, a prediction-mimicking knowledge distillation method for object detectors that feeds student intermediate head features through the frozen teacher head so the distillation targets avoid conflicting with ground-truth assignments.
Problem
Prediction-mimicking knowledge distillation for object detection suffers from target conflict: the teacher's predictions often disagree with the ground-truth targets assigned to the student, and prior methods down-weight those regions, which hurts performance.
Approach
Deliver the student's intermediate head features into the teacher's subsequent head layers to form cross-head predictions, and apply the distillation loss between these cross-head predictions and the teacher's original predictions, so the student's own head receives only detection loss plus a consistent distillation signal.
Key results
- On COCO, CrossKD raises GFL ResNet-50 student (1x schedule) from 40.2 to 43.7 AP using only prediction-mimicking losses, outperforming LD (43.0) and PKD (43.3).
- Combining CrossKD with PKD reaches 43.9 AP on GFL.
- CrossKD improves RetinaNet, FCOS and ATSS students by 2.3, 2.8 and 2.4 AP respectively, and with heterogeneous backbones improves RetinaNet MobileNetv2 from 30.9 to 34.1 AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CrossKD instead of direct prediction mimicking (ablation) | GFL-R18 · COCO minival | AP: 37.8 -> 38.7 | Helped |
| Use the third intermediate feature for CrossKD (ablation) | GFL-R18 · COCO minival | AP: 35.8 -> 38.7 | Helped |
| Apply CrossKD to classification and regression branches (ablation) | GFL-R18 · COCO minival | AP: 37.8 -> 38.7 | Helped |
| Add prediction mimicking with LD alongside CrossKD (ablation) | GFL-R18 · COCO minival | AP: 38.7 -> 38.1 | Hurt |
| Combine CrossKD with PKD feature imitation (ablation) | GFL-R50 · COCO minival | AP: 43.7 -> 43.9 | Helped |
| Apply CrossKD to RetinaNet | RetinaNet-R50 · COCO minival | AP: 37.4 -> 39.7 | Helped |
| Apply CrossKD to FCOS | FCOS-R50 · COCO minival | AP: 38.5 -> 41.3 | Helped |
| Apply CrossKD to ATSS | ATSS-R50 · COCO minival | AP: 39.4 -> 41.8 | Helped |
| Use CrossKD with a RetinaNet teacher despite different label assigners | GFL-R50 · COCO minival | AP: 40.2 -> 41.2 | Helped |
| Use direct prediction mimicking with a RetinaNet teacher | GFL-R50 · COCO minival | AP: 40.2 -> 30.3 | Hurt |
| Apply CrossKD with a Swin-T teacher | RetinaNet-ResNet-50 · COCO minival | AP: 36.5 -> 38.0 | Helped |
| Apply CrossKD with a MobileNetv2 student | RetinaNet-MobileNetv2 · COCO minival | AP: 30.9 -> 34.1 | Helped |
Surprising findings
- Combining conventional prediction mimicking with CrossKD lowers AP (38.1) compared with CrossKD alone (38.7), attributed to reintroduced target conflict.
- Under severe target conflict (RetinaNet teacher for GFL student), conventional KD drops AP to 30.3 while CrossKD reaches 41.2.
Limitations
- Gains are evaluated mainly on COCO with GFL-style detectors and ResNet backbones; generality beyond 2D dense detectors is not shown.
- Choice of insertion layer (i=3) was tuned on the same GFL R50/R18 setup used for ablations.
Experimental setup
Datasets: MS COCO trainval135k
Benchmarks: MS COCO minival
Metrics: AP AP50 AP75 APS APM APL
Baselines: FitNets Inside GT Box Defeat Main Region (LD) Fine-Grained FGD GID SKD LD PKD
Builds on: GFL FitNet Localization Distillation PKD MMDetection
Closest papers by meaning
- Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation (AAAI 2022 · similarity 0.92)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.88)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.88)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.87)
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.