Research map / Label-efficient and robust learning / Knowledge distillation
DFD: Distilling the Feature Disparity Differently for Detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Disparity Feature Distillation (DFD), which splits feature maps into high- and low-disparity regions by spatial-attention L1 distance and applies a weak learnable-transform constraint to high-disparity regions and a strict MSE constraint to low-disparity regions for detector knowledge distillation.
Problem
Feature distillation for object detectors typically imposes uniform imitation constraints, ignoring that the student-teacher feature disparity varies spatially; strict constraints can be hard to learn while weak constraints risk lazy students.
Approach
Partition feature regions by the mean of the spatial-attention L1 disparity map between teacher and student, then apply a lenient learnable transformation-module constraint to high-disparity regions and strict MSE to low-disparity regions, combining leniency and strictness.
Key results
- On COCO with RetinaNet ResNet50 student and ResNeXt101 teacher, DFD raises mAP from 37.4 to 41.7 (+4.0), outperforming FGD, MGD and PKD (40.9–41.2).
- Faster R-CNN ResNet50 improves from 38.4 to 42.4 mAP and RepPoints ResNet50 from 38.6 to 42.7 mAP.
- On YOLOv6-Tiny, mAP improves from 40.6 to 41.7 over the student baseline, surpassing PKD (41.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DFD feature distillation | RetinaNet ResNet50 · COCO2017 | mAP: 37.4% to 41.7% | Helped |
| apply DFD feature distillation | Faster R-CNN ResNet50 · COCO2017 | mAP: 38.4% to 42.4% | Helped |
| apply DFD feature distillation | RepPoints ResNet50 · COCO2017 | mAP: 38.6% to 42.7% | Helped |
| apply DFD feature distillation | Mask R-CNN Swin-Tiny · COCO2017 | mAP: 42.7 to 44.4 | Helped |
| apply DFD feature distillation | YOLOv6-Tiny · COCO2017 | mAP: 40.6 to 41.7 | Helped |
| apply DFD feature distillation | Heatmap MobileNetV2 · COCO-Body pose estimation | mAP: 62.0 to 62.6 | Helped |
| apply DFD feature distillation | PSPNet Res18 · Cityscapes segmentation | mAP: 69.85 to 73.74 | Helped |
| distill only high-disparity regions instead of the entire feature map (ablation) | RetinaNet ResNet50 · COCO | mAP: 36.4 -> 39.8 | Helped |
| add low-disparity-region distillation to high-disparity-region distillation (ablation) | RetinaNet ResNet50 · COCO | mAP: 39.8 -> 39.6; 39.6 -> 40.0 | Mixed |
| use weaker constraints in high-disparity regions and strict MSE in low-disparity regions (ablation) | RetinaNet ResNet50 · COCO | mAP: 40.0 -> 40.4 | Helped |
| remove MSE constraints from low-disparity regions (ablation) | RetinaNet ResNet50 · COCO | mAP: 40.4 to 39.9 | Hurt |
| swap the high- and low-disparity region constraints (ablation) | RetinaNet ResNet50 · COCO | mAP: 40.4 to 39.9 | Hurt |
Surprising findings
- Distilling only high-disparity regions outperformed distilling the whole feature map without differentiation (39.8 vs 39.6 mAP baseline comparisons in Table 1 ablation).
- Using MGD or PKD as the strong constraint (instead of weak) brought no improvement, supporting the leniency-for-hard-regions design.
Limitations
- Relies on two loss weight hyperparameters (alpha, beta) that must be tuned differently for one-stage and two-stage detectors.
- Region partition depends on a mean threshold over the L1 attention disparity map, a heuristic choice compared only against median and a fixed-threshold variant.
Experimental setup
Datasets: COCO2017 COCO-Body Cityscapes
Benchmarks: COCO2017 COCO-Body Cityscapes
Metrics: mAP
Baselines: FGD MGD PKD DiffKD CTFD CWD Defeat
Builds on: FitNets FGD MGD PKD DiffKD
Closest papers by meaning
- PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient (NeurIPS 2022 · similarity 0.92)
- Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors (ICLR 2021 · similarity 0.92)
- Focal and Global Knowledge Distillation for Detectors (CVPR 2022 · similarity 0.91)
- Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation (AAAI 2022 · similarity 0.91)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.89)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.