Research map / Label-efficient and robust learning / Recognition and long-tail

Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

AAAI 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DISCO, a training-free pre-trained model assessment method that weights truncated-SVD spectral components of extracted features by their task-specific transferability and singular-value ratios, applicable to classification and regression (object detection).

Problem

Selecting the best pre-trained model from a model hub for transfer learning without costly fine-tuning; existing transferability metrics rely on entire extracted features or class-specific hypotheses that limit them to classification.

Approach

Decompose extracted features via SVD into spectral components, observe that fine-tuning concentrates singular-value mass on more transferable components, and score each model by summing per-component task performance (nearest-centroid confidence for classification, linear-approximation MSE for regression) weighted by each component's singular-value ratio.

Key results

• On 11 image classification datasets with 11 supervised CNNs, DISCO reaches average weighted Kendall's tau 0.739, versus 0.726 for NCTI (second best).
• On 10 self-supervised ResNet-50 models, DISCO achieves average weighted Kendall's tau 0.726, the best among compared metrics.
• On object detection (5 datasets, 6 YOLO models), DISCO achieves average weighted tau 0.634 and Pr(top1)=1.000, about 21% above ETran's 0.522.

Measured effects reported in the paper

Surprising findings

• Components with larger singular values generally undergo the least change during fine-tuning, while variation peaks for mid-to-small components.
• Hard-example selection at 40% of data retains most accuracy (tau 0.730) while reducing time versus the full dataset.

Limitations

• Object detection evaluated on only six models and five datasets, a small pool.
• Full-dataset SVD cost grows with sample count; hard-example selection introduces a time-performance trade-off.
• Group number G must be chosen (G=6 or 8 recommended), and results vary across datasets, e.g. Pets and Aircraft are weaker.

Experimental setup

Datasets: FGVC Aircraft Stanford Cars Food101 Oxford-IIIT Pets Oxford 102 Flowers Caltech101 CIFAR-10 CIFAR-100 VOC2007 SUN397 DTD Blood

Benchmarks: FGVC Aircraft Caltech101 Stanford Cars CIFAR-10 CIFAR-100 DTD Oxford 102 Flowers Food101 Oxford-IIIT Pets SUN397 VOC2007 Blood

Metrics: weighted Kendall's tau top-k probability Pr(top-k)

Baselines: NCE LEEP LogME NLEEP SFDA ETran NCTI PACTran Linear

Builds on: LogME SFDA NCTI ETran LEEP

Closest papers by meaning

• Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.85)
• Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.83)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.83)
• Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.83)
• Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.83)
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.83)