Research map / Label-efficient and robust learning / Recognition and long-tail
Understanding the Transferability of Representations via Task-Relatedness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an upper bound on transfer-learning transferability in a cross-domain cross-task setting, decomposed into re-weighted reference loss, label mismatch (conditional entropy) and Wasserstein distribution mismatch, and uses the resulting "task-relatedness" as a transferability estimation metric.
Problem
When can representations of a pre-trained encoder transfer to a downstream target task whose features, class priors and label sets may all differ from the reference task, and how can transferability be estimated cheaply for model selection?
Approach
Transform the reference task's class prior, label set (via a conditional matrix B) and feature space (via an invertible linear map A) to match the target, yielding a provable upper bound on target cross-entropy loss as the sum of three interpretable terms; task-relatedness is this sum and can be estimated with pseudo-labels or mean/covariance matching.
Key results
- Task-relatedness achieves Pearson correlation of -0.82 on average with end-to-end fine-tuning accuracy across Pets, DTD, and Aircraft, versus 0.90 for LEEP and 0.37 for LogMe.
- Using pseudo-labels instead of true target labels keeps correlation for Pets (-0.76 vs -0.77) and DTD (-0.91 vs -0.97), but drops sharply for Aircraft (-0.16 vs -0.72).
- Learning transformations takes 3-4 minutes, versus almost a day for full fine-tuning of one model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-relatedness to estimate transferability from reference-task and target-task data | Pre-trained model selection · Pets | Pearson correlation with fine-tuning accuracy: -0.77 vs 0.88 | Mixed |
| Use task-relatedness to estimate transferability from reference-task and target-task data | Pre-trained model selection · DTD | Pearson correlation with fine-tuning accuracy: -0.97 vs 0.96 | Mixed |
| Use task-relatedness to estimate transferability from reference-task and target-task data | Pre-trained model selection · Aircraft | Pearson correlation with fine-tuning accuracy: -0.72 vs 0.97 | Mixed |
| Use pseudo labels instead of true target labels (ablation) | Task-relatedness · Pets | Pearson correlation with fine-tuning accuracy: -0.77 -> -0.76 | No clear effect |
| Use pseudo labels instead of true target labels (ablation) | Task-relatedness · DTD | Pearson correlation with fine-tuning accuracy: -0.97 -> -0.91 | Hurt |
| Use pseudo labels instead of true target labels (ablation) | Task-relatedness · Aircraft | Pearson correlation with fine-tuning accuracy: -0.72 -> -0.16 | Hurt |
| Replace Wasserstein distance with mean and covariance matching (ablation) | Task-relatedness estimation · Target-task transferability estimation | computation efficiency | Helped |
Surprising findings
- Aircraft pseudo-label correlation drops from -0.72 to -0.16, showing unsupervised estimation fails when the reference classifier transfers poorly.
- Several SbTE metrics (e.g., NCE, LogMe) produce high negative correlations on some targets, indicating instability.
Limitations
- Analysis restricted to cross-entropy loss; 0-1 loss not covered.
- Wasserstein distance is hard to compute with limited samples in high-dimensional representation spaces; mean/covariance surrogate used in practice.
- Requires Lipschitz constraint on loss-classifier composition, which restricts the hypothesis class.
Experimental setup
Datasets: ImageNet DBPedia CIFAR10 CIFAR100 Aircraft Pets DTD AG-News Yelp-5 SST-5 MNIST Fashion-MNIST
Benchmarks: CIFAR10 CIFAR100 Aircraft Pets DTD AG-News Yelp-5 SST-5 MNIST USPS MNIST-M
Metrics: Pearson correlation transferability bound gap task-relatedness value
Baselines: LogMe LEEP NCE PACTran SFDA H-Score OT-NCE OTCE
Builds on: domain adaptation bounds (Ben-David et al.) NCE OTCE PACTran
Closest papers by meaning
- Understanding Task Transfer in Vision-Language Models (CVPR 2026 · similarity 0.90)
- Implicit Modeling for Transferability Estimation of Vision Foundation Models (NeurIPS 2025 · similarity 0.88)
- Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings (NeurIPS 2025 · similarity 0.88)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.88)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.87)
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.