AI Paper Atlas

Research map / Label-efficient and robust learning / Domain adaptation and generalization

Distilling from Similar Tasks for Transfer Learning on a Budget

Kenneth Borup, Cheng Perng Phoo, Bharath Hariharan

ICCV 2023 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Geometric alignment and synthetic-to-real adaptation · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill from the most similar source model instead of fine-tuning from ImageNetMobileNetV3 · 8 target tasks (mean)top-1 accuracy (%): 67.2 -> 72.2Improved
weight multiple source models by task similarity instead of one sourceDistillNearest · 8 target tasks (mean)top-1 accuracy (%): 72.2 -> 72.8Improved
choose the source model by task similarity instead of randomly (ablation)single-source distillation · 8 target tasks (mean)top-1 accuracy (%): 67.8 -> 72.2Improved
weight sources by similarity instead of equal weights (ablation)multi-source distillation · 8 target tasks (mean)top-1 accuracy (%): 69.3 -> 72.8Improved
use random source weights instead of similarity weights (ablation)multi-source distillation · 8 target tasks (mean)top-1 accuracy (%): 72.8 -> 67.3Hurt
invert the similarity ranking for source weights (ablation)multi-source distillation · 5 target taskstop-1 accuracyHurt
sharpen similarity weights with exponent p=12 instead of p=1 (ablation)multi-source distillation · 8 target taskstop-1 accuracyImproved
use feature representations rather than pseudo-label outputs for task similarity (ablation)PARC similarity · 8 target tasksSpearman correlation: 0.45 -> 0.58Improved
add FixMatch with unlabeled data to ImageNet fine-tuningMobileNetV3 · 8 target tasks (mean)top-1 accuracy (%): 67.2 -> 68.3Mixed
apply distillation to structured VTAB tasksmulti-source distillation · VTAB Structuredtop-1 accuracy (%): 49.1 -> 43.7Hurt
apply multi-source distillation on natural VTAB tasksMobileNetV3 · VTAB Naturaltop-1 accuracy (%): 66.3 -> 71.4Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy