Research map / Label-efficient and robust learning / Knowledge distillation
Harmonizing knowledge Transfer in Neural Network with Unified Distillation
yaomin huang, Faming Fang, Zaoming Yan, Chaomin Shen, Guixu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Feature Distribution Prediction (FDP) module (ablation) | UniKD · CIFAR-100 ResNet110 -> ResNet32 | top-1 accuracy: 73.08 -> 74.42 | Improved |
| add Adaptive Features Fusion (AFF) on top of FDP (ablation) | UniKD · CIFAR-100 ResNet110 -> ResNet32 | top-1 accuracy: 74.42 -> 75.36 | Improved |
| add FDP (ablation) | UniKD · CIFAR-100 WRN-40-2 -> ShuffleNet-V1 | top-1 accuracy: 75.21 -> 77.10 | Improved |
| add AFF on top of FDP (ablation) | UniKD · CIFAR-100 WRN-40-2 -> ShuffleNet-V1 | top-1 accuracy: 77.10 -> 78.10 | Improved |
| unify features and logits into one distribution-level KL constraint instead of directly hybridizing feature and logits KD | UniKD · CIFAR-100 six teacher-student pairs | average top-1 accuracy: 74.95 -> 75.89 | Improved |
| UniKD versus MLKD+KR hybrid | UniKD · CIFAR-100 WRN-40-2 -> WRN-16-2 | top-1 accuracy: 77.54 -> 77.49 | No clear effect |
| distribution-level constraints for dense prediction (object detection) | UniKD · MS-COCO Faster-RCNN FPN | AP: 37.01 -> 37.24 (ResNet101->ResNet18 vs KR+DKD) | Mixed |
| UniKD distillation on ImageNet vs previous SOTA | UniKD · ImageNet ResNet34 -> ResNet18 | top-1 accuracy: 71.70 -> 72.83 | Improved |
| UniKD distilled student versus teacher | UniKD · CIFAR-100 six teacher-student pairs | average top-1 accuracy vs teacher: +0.54 over teacher | Improved |
Closest papers by meaning
- Heterogeneous Knowledge Distillation Using Information Flow Modeling (CVPR 2020 · similarity 0.91)
- Variational Information Distillation for Knowledge Transfer (CVPR 2019 · similarity 0.91)
- Direction Sensitivity–Based Knowledge Distillation: Optimization-Aware Low-Rank Knowledge Transfer (AAAI 2026 · similarity 0.91)
- Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (ECCV 2020 · similarity 0.91)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
