Research map / Label-efficient and robust learning / Knowledge distillation
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that arbitrary pretrained ImageNet models hold complementary knowledge for each other and proposes KL distillation with confidence-based data partitioning (KL+DP) to transfer it without performance degradation, raising transfer success from under 40% to 92.5%.
Problem
Transferring complementary knowledge between arbitrary already-trained models (including weaker or equal-performing teachers) without degrading the student's existing performance, lacking external test rankings.
Approach
Treat transfer between pretrained models as continual learning and partition transfer data per sample by teacher-vs-frozen-student confidence, distilling from the teacher only on samples where it is more confident and from the frozen student elsewhere; this needs no extra hyperparameters and can be done without labels.
Key results
- KL+DP raises median transfer success rate to 92.5% vs under 40% for normal KL distillation (32.5% success rate for KL-Dist per introduction).
- On full ImageNet, KL-Dist+DP gives positive transfer deltas for all eight students (e.g., XCiT-P16 +0.65, PiT-B +0.55) where KL-Dist is mixed (e.g., ConvNext -0.51, Mixer-B16 -0.29 for vanilla KL).
- Sequential transfer from three teachers gives a 59% average gain over the best single teacher (e.g., PiT-B 0.86 to 1.04).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use data partitioning instead of standard KL distillation | pretrained ImageNet students · ImageNet | transfer success rate: 32.5% -> 92.5% | Helped |
| Add momentum weight interpolation (MCL) to XE-KL distillation | pretrained ImageNet students · 10% ImageNet subset | transfer success rate: 80% | Helped |
| Replace standard distillation with confidence-based data partitioning | pretrained ImageNet students · ImageNet | transfer delta: 0.13 ($\pm$1.01) -> 0.65 ($\pm$0.26); 0.33 ($\pm$0.86) -> 0.55 ($\pm$0.25); 0.10 ($\pm$0.49) -> 0.40 ($\pm$0.12); -0.05 | Helped |
| Use unsupervised confidence-based selection instead of supervised data partitioning (ablation) | KL+DP transfer · ImageNet | transfer delta: 0.65 ($\pm$0.26) -> 0.96 ($\pm$0.42); 0.55 ($\pm$0.25) -> 0.95 ($\pm$0.52); 0.40 ($\pm$0.12) -> 0.35 ($\pm$0.18); 0.27 ( | No clear effect |
| Remove the auxiliary cross-entropy classification objective from KL+DP (ablation) | KL+DP transfer · ImageNet transfer pairs | transfer performance | Helped |
| Replace KL divergence with cross-entropy loss (ablation) | distillation transfer · 60 teacher-student pairs | transfer delta: 1.2 percentage points lower | Hurt |
| Add a warmup epoch training only the student's linear layers (ablation) | KL distillation transfer · teacher-student transfer pairs | transfer performance | No clear effect |
| Restrict the KL loss to the top-10 teacher-predicted classes (ablation) | distillation transfer · six teacher-student pairs | transfer delta: 0.20 percentage points | Helped |
| Use contrastive distillation (CD or CRD) instead of KL-based transfer (ablation) | pretrained expert models · 10% ImageNet subset | transfer success rate | Hurt |
| Transfer sequentially from multiple teachers instead of the best single teacher | KL+DP students · ImageNet | transfer delta: 0.26 -> 0.65; 0.86 -> 1.04 | Helped |
| Transfer from multiple teachers in parallel instead of the best single teacher | KL+DP students · ImageNet | transfer delta | No clear effect |
| Average separately distilled student variants into a model soup | KL+DP students · ImageNet | transfer delta | Hurt |
Surprising findings
- Unsupervised confidence-based data partitioning matches or sometimes outperforms supervised partitioning (e.g., PiT-B 0.95 unsupervised vs 0.55 supervised).
- Complementary knowledge exists even when the teacher is up to 20% weaker than the student, and weaker teachers' knowledge is more concentrated in specific classes.
Limitations
- Gains fall short of total complementary knowledge due to retention-transfer trade-off.
- Confidence-based unsupervised partitioning has higher variance and can be overconfident.
- Experiments limited to ImageNet-scale image classification with a constrained 20-epoch budget and 10% subset for exploration.
Experimental setup
Datasets: ImageNet CUB200 StanfordCars Caltech256
Benchmarks: ImageNet CUB200 StanfordCars Caltech256
Metrics: top-1 accuracy transfer delta success rate (share of teachers with positive transfer) positive/negative prediction flips
Baselines: KL-Dist XE-KL XE-KL+MCL label smoothing CD distillation CRD distillation Model soups
Builds on: knowledge distillation (Hinton et al.) momentum continual learning (MCL) model soups CRD contrastive distillation ffcv
Closest papers by meaning
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.89)
- Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model (AAAI 2021 · similarity 0.88)
- Deep Model Transferability from Attribution Maps (NeurIPS 2019 · similarity 0.88)
- MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · similarity 0.86)
- GraphBridge: Towards Arbitrary Transfer Learning in GNNs (ICLR 2025 · similarity 0.86)
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.