Research map / Vision-language and multimodal / CLIP and zero-shot
Transferring Knowledge From Large Foundation Models to Small Downstream Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Feature Transfer (AFT), which regularizes a small downstream model's features via a kernel-distance term against frozen pre-trained features with learned diagonal weights, transferring task-relevant knowledge from large foundation models without weight transfer.
Problem
Transferring knowledge from large foundation models into small task-specific downstream models, which standard weight-initialization transfer does not do well and which cannot combine multiple pre-trained models with different architectures.
Approach
Define a prior favoring low mutual information between downstream features and inputs conditioned on pre-trained features; approximate it by a kernel-distance regularizer between downstream kernels and learned diagonal-rescaled pre-trained kernels, so the downstream model adaptively learns only the pre-trained features relevant to the task.
Key results
- AFT reduces error by an average of over 15% relative to STL when transferring from DINOv2 ViT-G/14 to ViT-S, MLP-Mixer-B and ResNet-50 across six vision datasets.
- AFT achieves the highest correlation (0.97) between downstream accuracy and pre-trained linear probe accuracy, versus other methods that saturate early.
- On SNLI-VE, AFT with CLIP ResNet-50 downstream reaches 74.39 accuracy vs 73.69 for STL and 74.05 for KD, transferring from DINOv2 and LLaMA 13B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer features from DINOv2 ViT-L/14 and LLaMA 13B with AFT | CLIP ResNet-50 · SNLI-VE | accuracy: $73.69_{\pm0.28}$ -> $74.39_{\pm0.18}$ | Helped |
| Use the kernel formulation rather than the ℓ2 objective (ablation) | AFT · CIFAR-100 and BoolQ | performance | Helped |
| Learn feature weights μ rather than fixing μ to identity (ablation) | AFT · CIFAR-100 and BoolQ | performance | Helped |
| Use dense rather than diagonal μ (ablation) | AFT · CIFAR-100 and BoolQ | performance | Hurt |
| Replace the linear kernel with an RBF kernel (ablation) | AFT · CIFAR-100 and BoolQ | performance | No clear effect |
| Use bi-level optimization with 5 inner μ updates per θ update (ablation) | AFT · CIFAR-100 and BoolQ | performance | No clear effect |
| Combine features from multiple pre-trained vision models | AFT · CIFAR-100 | accuracy: $88.88_{\pm0.06}$ -> $89.23_{\pm0.10}$ | Helped |
| Add random noise features to the transferred features | AFT with DINOv2 ViT-G/14 features · CIFAR-100 | performance | No clear effect |
| Use AFT-learned weights μ on pre-trained features | Linear probe on CIFAR-100 features · CIFAR-100 | linear probe error | Helped |
| Transfer from stronger pre-trained models using AFT | AFT · Vision benchmarks | correlation with pre-trained linear probe accuracy: 0.97 | Helped |
| Use AFT rather than STL | ViT-S, MLP-Mixer-B, and ResNet-50 · CIFAR-10, CIFAR-100, Oxford Flowers-102, Oxford-IIIT Pets, DTD, and Food-101 | error relative to STL: over 15% | Helped |
Surprising findings
- Combining multiple pre-trained language models often does not improve linear probe or AFT accuracy, suggesting little complementary information among them.
- On ResNet-50, KD slightly outperforms AFT in most cases.
Limitations
- Downstream model capacity bounds performance; small models limit achievable gains.
- Only last-layer features are transferred in the current formulation.
- Language experiments show little gain from combining multiple pre-trained models.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Oxford Flowers-102 Oxford-IIIT Pets DTD Food-101 IMDB BoolQ MNLI SST-2 MRPC
Benchmarks: CIFAR-10 CIFAR-100 Oxford Flowers-102 Oxford-IIIT Pets DTD Food-101 IMDB BoolQ MNLI SST-2 MRPC QQP
Metrics: accuracy normalized error linear probe accuracy inference time Pearson correlation
Baselines: Standard transfer learning (STL) B-Tuning knowledge distillation (KD) relational knowledge distillation (RKD) factor transfer (FT)
Builds on: B-Tuning knowledge distillation relational knowledge distillation factor transfer
Closest papers by meaning
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.87)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.86)
- Implicit Modeling for Transferability Estimation of Vision Foundation Models (NeurIPS 2025 · similarity 0.86)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.86)
- Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains (NeurIPS 2025 · similarity 0.85)
- Update Your Transformer to the Latest Release: Re-Basin of Task Vectors (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.