Research map / Label-efficient and robust learning / Domain adaptation and generalization
Distilling from Similar Tasks for Transfer Learning on a Budget
Kenneth Borup, Cheng Perng Phoo, Bharath Hariharan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from the most similar source model instead of fine-tuning from ImageNet | MobileNetV3 · 8 target tasks (mean) | top-1 accuracy (%): 67.2 -> 72.2 | Improved |
| weight multiple source models by task similarity instead of one source | DistillNearest · 8 target tasks (mean) | top-1 accuracy (%): 72.2 -> 72.8 | Improved |
| choose the source model by task similarity instead of randomly (ablation) | single-source distillation · 8 target tasks (mean) | top-1 accuracy (%): 67.8 -> 72.2 | Improved |
| weight sources by similarity instead of equal weights (ablation) | multi-source distillation · 8 target tasks (mean) | top-1 accuracy (%): 69.3 -> 72.8 | Improved |
| use random source weights instead of similarity weights (ablation) | multi-source distillation · 8 target tasks (mean) | top-1 accuracy (%): 72.8 -> 67.3 | Hurt |
| invert the similarity ranking for source weights (ablation) | multi-source distillation · 5 target tasks | top-1 accuracy | Hurt |
| sharpen similarity weights with exponent p=12 instead of p=1 (ablation) | multi-source distillation · 8 target tasks | top-1 accuracy | Improved |
| use feature representations rather than pseudo-label outputs for task similarity (ablation) | PARC similarity · 8 target tasks | Spearman correlation: 0.45 -> 0.58 | Improved |
| add FixMatch with unlabeled data to ImageNet fine-tuning | MobileNetV3 · 8 target tasks (mean) | top-1 accuracy (%): 67.2 -> 68.3 | Mixed |
| apply distillation to structured VTAB tasks | multi-source distillation · VTAB Structured | top-1 accuracy (%): 49.1 -> 43.7 | Hurt |
| apply multi-source distillation on natural VTAB tasks | MobileNetV3 · VTAB Natural | top-1 accuracy (%): 66.3 -> 71.4 | Improved |
Closest papers by meaning
- Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data (NeurIPS 2021 · similarity 0.89)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.86)
- DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation (ICLR 2025 · similarity 0.86)
- Revisiting Semi-Supervised Learning in the Era of Foundation Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
