Research map / Label-efficient and robust learning / Knowledge distillation
DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCSF-KD, a feature-based knowledge distillation method for object detection that weights per-channel spatial feature MSE by GAP-derived teacher channel attention, evaluated on COCO with homogeneous and heterogeneous teacher-student pairs.
Problem
Existing knowledge distillation methods for object detection treat spatial and channel knowledge independently and often require complex foreground/background weighting functions, causing redundancy and instability across detector architectures.
Approach
Combine channel-wise spatial feature distillation with dynamic channel attention: batch-normalized FPN features are weighted per channel by global-average-pooled teacher activations (softmax-constrained), so channels with higher activation contribute more to the distillation MSE loss, without designing spatial masks.
Key results
- On MS COCO, DCSF-KD raises RetinaNet-ResNet50 from 37.4 to 41.0 mAP with ResNeXt101 teacher, and FCOS-ResNet50 reaches 42.9 mAP with a 2x+ms schedule.
- With Mask R-CNN-Swin as teacher, DCSF-KD reaches 41.9 mAP for RetinaNet-ResNet50 and 44.1 mAP for FCOS-ResNet50.
- Ablation on RetinaNet pair: adaptive channel weights add 0.2 mAP over CKD and 0.8 over CWD-style baseline on RepPoints pair (42.4 vs 42.0 CWD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCSF-KD instead of PKD | RetinaNet-ResX101 → ResNet-50 · MS COCO | mAP: 40.8 -> 41.0 | Helped |
| Use DCSF-KD instead of PKD | Faster R-CNN-Res101 → Res50 · MS COCO | mAP: 40.5 -> 40.7 | Helped |
| Use DCSF-KD instead of PKD | FCOS-Res101 → Res50 · MS COCO | mAP: 42.8 -> 42.9 | Helped |
| Use DCSF-KD instead of PKD | RepPoints-ResNeXt101 → Res50 · MS COCO | mAP: 42.3 -> 42.4 | Helped |
| Use DCSF-KD instead of PKD | TOOD-ResX101 → Res50 · MS COCO | mAP: 46.0 -> 46.2 | Helped |
| Use DCSF-KD instead of CrossKD | GFL-R101 → R50 · MS COCO | mAP: 43.7 -> 44.1 | Helped |
| Use DCSF-KD instead of PKD | Mask R-CNN-Swin → RetinaNet-Res50 · MS COCO | mAP: 41.5 -> 41.9 | Helped |
| Use DCSF-KD instead of PKD | Mask R-CNN-Swin → FCOS-Res50 · MS COCO | mAP: 43.9 -> 44.1 | Helped |
| Use DCSF-KD instead of PKD | GFL-Res101 → FCOS-Res50 · MS COCO | mAP: 43.5 -> 43.7 | Helped |
| Add adaptive channel attention weights to CKD (ablation) | RetinaNet-ResX101 → ResNet-50 · MS COCO | mAP: 40.6 -> 41.0 | Helped |
| Add adaptive channel attention weights to CKD (ablation) | RepPoints-ResNeXt101 → Res50 · MS COCO | mAP: 41.6 -> 42.4 | Helped |
| Vary loss weight α (ablation) | DCSF-KD · MS COCO (RetinaNet and Faster R-CNN teacher-student pairs) | mAP: 0.2% decrease | No clear effect |
Surprising findings
- Loss weight alpha is insensitive, with only about 0.2% mAP drop in the worst setting.
Limitations
- Evaluated only on COCO and only on detection; extension to DETR-style and 3D detection or segmentation is future work.
- Only a single hyperparameter alpha, but separate values chosen for two-stage and single-stage models (2 vs 1).
Experimental setup
Datasets: MS COCO
Benchmarks: MS COCO
Metrics: mAP AP50 AP75 APS APM APL
Baselines: FKD FRS FGD PKD GID CrossKD CWD CKD
Builds on: CWD PKD FGD SENet AT
Closest papers by meaning
- PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient (NeurIPS 2022 · similarity 0.92)
- Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors (ICLR 2021 · similarity 0.91)
- Focal and Global Knowledge Distillation for Detectors (CVPR 2022 · similarity 0.90)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.89)
- CrossKD: Cross-Head Knowledge Distillation for Object Detection (CVPR 2024 · similarity 0.88)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.