Research map / Label-efficient and robust learning / Knowledge distillation
Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation
Guopeng Li, Qiang Wang, Ke Yan, Shouhong Ding, Yuan Gao, Gui-Song Xia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MSA module and S_m^4 stage to fused model (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 75.95 -> 81.61 | Improved |
| add S_m^4 stage to fused model (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 77.21 -> 81.61 | Improved |
| add teacher-to-fusion loss (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 25.57 -> 81.61 | Improved |
| add teacher-to-fusion loss (ablation) | FBT · CIFAR100 (ConvNeXt-T -> Swin-P) | top-1 accuracy (%): 50.46 -> 80.73 | Improved |
| add fusion-to-student loss (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 79.01 -> 81.61 | Improved |
| add direct teacher-to-student loss (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 79.26 -> 81.61 | Improved |
| add InfoNCE feature loss (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 80.95 -> 81.61 | Improved |
| add InfoNCE feature loss (ablation) | FBT · CIFAR100 (ConvNeXt-T -> Swin-P) | top-1 accuracy (%): 78.89 -> 80.73 | Improved |
| add OFA logit loss (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 77.91 -> 81.61 | Improved |
| add OFA logit loss (ablation) | FBT · ImageNet-1K (ResNet50 -> DeiT-T) | top-1 accuracy (%): 72.13 -> 75.64 | Improved |
| fuse CNN stages 1-3 with MSA stage 4 (ViT-S teacher) instead of stages 1-2 with MSA stage 3 (ablation) | FBT · CIFAR100 (ViT-S -> ResNet18) | top-1 accuracy (%): 82.3 -> 81.93 | Hurt |
| use fused architecture S_c^{1->3} -> S_m^{4->fc} instead of S_c^{1->3} -> S_m^{3->fc} (ablation) | FBT · CIFAR100 (Swin-T -> ResNet18) | top-1 accuracy (%): 78.56 -> 81.61 | Improved |
Closest papers by meaning
- Heterogeneous Complementary Distillation (AAAI 2026 · similarity 0.94)
- Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation (AAAI 2025 · similarity 0.91)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
