Research map / Label-efficient and robust learning / Knowledge distillation
FreeKD: Knowledge Distillation via Semantic Frequency Prompt
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeKD, a knowledge distillation method that transfers wavelet frequency bands rather than spatial features, using learnable Frequency Prompts to mask pixels of interest and a position-aware relational frequency loss for dense prediction.
Problem
Spatial feature imitation in knowledge distillation for dense prediction is corrupted by consecutive downsampling in the teacher, and naively distilling frequency bands mixes useful signal with noise, so the student needs to know which pixels in which frequency bands to imitate.
Approach
Insert learnable frequency prompts into the teacher (finetuned with a Jaccard-based dissimilarity loss) to generate pixel-wise masks over wavelet (DWT) frequency bands, and distill only the masked bands weighted by a position-aware channel gate built from teacher and student features.
Key results
- FreeKD improves RepPoints-R50 student to 42.4 AP with RepPoints-X101 teacher, surpassing DiffKD by 0.7 AP.
- FreeKD gives 4.4 AP gain for FCOS-R50 over the baseline student (42.9 vs 38.5) in the baseline setting on COCO.
- FreeKD achieves +0.77 mIoU for PSPNet-R18 and +0.43 mIoU for DeepLabV3-R18 gains over MGD on Cityscapes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply FreeKD instead of DiffKD | FCOS-R50 · COCO val2017 with R101 teacher | AP: 42.4 -> 42.9 | Helped |
| apply FreeKD instead of DiffKD | RepPoints-R50 · COCO val2017 with X101 teacher | AP: 41.7 -> 42.4 | Helped |
| apply FreeKD instead of MGD | PSPNet-R18 · Cityscapes val | mIoU: 73.63 -> 74.40 | Helped |
| apply FreeKD instead of DiffKD | RetinaNet-R50 · COCO-C | rPC: 51.1 -> 52.1 | Helped |
| add Frequency Prompt to low-frequency distillation (ablation) | RepPoints-R50 · COCO | AP: 40.7 -> 41.0 | Helped |
| add Frequency Prompt to high-frequency distillation (ablation) | RepPoints-R50 · COCO | AP: 41.8 -> 42.3 | Helped |
| add Frequency Prompt to combined low- and high-frequency distillation (ablation) | RepPoints-R50 · COCO | AP: 41.3 -> 42.4 | Helped |
| use low-frequency bands only instead of high-frequency bands (ablation) | RepPoints-R50 without Frequency Prompt · COCO | AP: 41.8 -> 40.7 | Hurt |
| replace position-aware weight with SE attention (ablation) | FreeKD on RepPoints-R50 · COCO | AP: 42.4 -> 42.1 | Hurt |
| replace DWT with DCT (ablation) | FreeKD on RepPoints-R50 · COCO | AP: 41.9 vs. 42.4 | Hurt |
| replace DWT with DFT (ablation) | FreeKD on RepPoints-R50 · COCO | AP: 42.0 vs. 42.4 | Hurt |
| add FreeKD | Deformable DETR-R18 · COCO | AP: 36.4 -> 38.9 | Helped |
Surprising findings
- Low-frequency-only distillation impairs performance when combined with high-frequency bands (-0.5 AP), showing naive frequency distillation is harmful without masking.
Limitations
- Frequency prompt requires a finetuning stage of the teacher before distillation.
- Gains on some settings are modest (e.g., 0.43 mIoU for DeepLabV3-R18).
Experimental setup
Datasets: COCO train2017 Cityscapes SA-1B COCO-C
Benchmarks: COCO val2017 Cityscapes val COCO-C SA-1B
Metrics: AP mIoU mPC rPC mAP clean
Baselines: FRS FGD DiffKD MGD CWD FKD MSE
Builds on: FGD DiffKD MGD Visual Prompt Tuning
Closest papers by meaning
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.86)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.86)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.