Research map / Architectures and efficiency / Pruning and efficient tuning

Implicit Modeling for Transferability Estimation of Vision Foundation Models

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Efficient ViT adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Implicit Transferability Modeling (ITM) with a Divide-and-Conquer Variational Approximation to estimate fine-tuned performance ranking of diverse vision foundation models without full fine-tuning.

Problem

Transferability estimation for model selection struggles to generalize across emerging pre-trained vision models with diverse architectures (CNN, ViT), pre-training strategies (supervised, contrastive, MIM), and task heads such as segmentation.

Approach

Implicitly model each model-task transferability as a latent variable z injected via a learnable mapping into mini-batch embedding subspaces; approximate post-fine-tuning embeddings by pulling batches toward pseudo-cluster centers using a closed-form deparametric update, so no per-batch inner optimization is needed.

Key results

• ITM achieves average weighted Kendall's tau_w of 0.61 across 10 downstream datasets with 10 pre-trained models, versus 0.45 for the best baseline (LogME).
• ITM takes 8.42 seconds on average (excluding 738 seconds of feature extraction), compared with 1.93 s for LogME and 3.58 s for ETran.
• Over five random seeds ITM scores 0.60 ± 0.01; on a 14-CNN benchmark ITM reaches 0.624 average tau_w, the best among compared methods.

Measured effects reported in the paper

Surprising findings

• Static and dynamic TE methods designed for CNNs fail to generalize to MIM-pretrained ViTs, with PARC and ETran failing on MAE.
• Different pseudo-cluster center generation strategies (PCA, Laplacian, one-hot) yield minimal performance variation, attributed to the linear classification head's invariance to orthogonal transforms.

Limitations

• Relies on final embedding features and cannot directly handle detection or vision-language supervision.
• Not evaluated under parameter-efficient fine-tuning (PEFT) protocols.
• Segmentation evaluation is a small proxy benchmark with only five models.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 FGVC Aircraft Caltech-101 DTD Oxford-IIIT Pets Stanford Cars SUN397 Food-101 Oxford 102 Flowers CamVid Cityscapes

Benchmarks: Caltech-101 Stanford Cars CIFAR-100 CIFAR-10 DTD FGVC Aircraft Oxford 102 Flowers Food-101 Oxford-IIIT Pets SUN397 CamVid Cityscapes

Metrics: weighted Kendall's tau wall-clock time Spearman's rho Kendall's tau mIoU

Baselines: NLEEP LogME PARC SFDA ETran PED SA (LDA) LEAD

Builds on: LEEP NCE LogME PED SFDA LEAD

Closest papers by meaning

• Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.88)
• Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.86)
• Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.86)
• Understanding Task Transfer in Vision-Language Models (CVPR 2026 · similarity 0.86)
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.85)
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses (NeurIPS 2025 · similarity 0.85)