Research map / Detection and segmentation / CNN features
Knowledge Diffusion for Distillation
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Chang Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion denoising of student feature before MSE distillation | MSE feature KD · ImageNet R18 (R34) | Top-1 (%): 70.58 -> 72.22 | Improved |
| add diffusion denoising of student feature before MSE distillation | MSE feature KD · ImageNet MobileNetV1 (R50) | Top-1 (%): 72.39 -> 73.62 | Improved |
| replace KL loss with DIST loss in DiffKD logits | DiffKD · ImageNet R18 (R34) | Top-1 (%): 72.22 -> 72.49 | Improved |
| use DiffKD instead of DIST | DIST · ImageNet strong strategies, MobileNetV2 | Top-1 (%): 74.4 -> 74.9 | Improved |
| use DiffKD instead of FGD on COCO | FGD · COCO RetinaNet-X101 teacher | AP: 40.4 -> 40.7 | Improved |
| use DiffKD instead of MasKD | MasKD · Cityscapes DeepLabV3-R18 | mIoU (%): 77.00 -> 77.78 | Improved |
| add attention-based MSE loss to DiffKD (ablation) | DiffKD · COCO detection | AP: +~0.1 AP (RetinaNet stronger teacher 40.7 -> 41.4) | Improved |
| add adaptive noise matching (ablation) | DiffKD · ImageNet MobileNetV1 | Top-1 (%): 73.34 -> 73.62 | Improved |
| distill both feature and logits instead of feature only (ablation) | DiffKD · ImageNet MobileNetV1 | Top-1 (%): 73.16 -> 73.62 | Improved |
| add linear autoencoder with 1024 dims (ablation) | DiffKD · ImageNet MobileNetV1 | Top-1 (%): 73.47 -> 73.62 | Improved |
| compress autoencoder to 128 dims (ablation) | DiffKD · ImageNet MobileNetV1 | Top-1 (%): 73.47 -> 72.84 | Hurt |
| increase denoising steps from 1 to 5 NFEs (ablation) | DiffKD · ImageNet MobileNetV1 | Top-1 (%) | Improved |
Closest papers by meaning
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.89)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.89)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.88)
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.88)
- Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
