Research map / Label-efficient and robust learning / Self-supervised and contrastive
Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes task-oriented knowledge transfer that adapts a vision foundation model (VFM) to a target task, distills its task predictions into a small model on an unlabeled transfer set, then finetunes, plus retrieval-augmented transfer set curation.
Problem
Training small task-specific vision models with limited labeled data is hard; large VFMs are too expensive to deploy, and web-scale or ImageNet/CLIP/DINO pretraining is compute-intensive.
Approach
Teach the target task to a VFM first, then distill task-level predictions (not generic features) from the adapted VFM into a small target model on an unlabeled transfer set, followed by finetuning; transfer sets can be generic web data (CC3M) or retrieved from a web-scale gallery with query-balanced crop-level retrieval.
Key results
- Task-oriented transfer outperforms task-agnostic VFM distillation, CLIP, ImageNet and DINO pretraining by up to 11.6%, 22.1%, 13.7% and 29.8% across tasks (generic and task-related transfer sets).
- Task-oriented transfer achieves up to 9x, 4x and 15x reduction in pretraining compute versus task-agnostic distillation, ImageNet pretraining and DINO pretraining, while outperforming them; CLIP pretraining used about 43K GPUh.
- Retrieval-augmented transfer sets (e.g., 154K ADE20K images) clearly outperform the 2.87M-image CC3M transfer set for ADE20K segmentation at all labeled-data sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace generic CC3M transfer with a 154K retrieval-augmented transfer set | Task-oriented transfer from DINOv2 to FastViT · ADE20K (4802 labeled images) | mean IoU: 41.44 -> 43.28 | Helped |
| Replace generic CC3M transfer with a 51K retrieval-augmented transfer set | Task-oriented transfer from OpenCLIP to FastViT · EuroSAT (100 labeled images) | accuracy: 90.21 -> 93.25 | Helped |
| Use task-oriented rather than task-agnostic VFM transfer | FastViT with OpenCLIP · ImageNet (50% labeled training images) | accuracy: 77.97 -> 81.43 | Helped |
| Use task-oriented rather than task-agnostic VFM transfer | FastViT with OpenCLIP · EuroSAT (10 labeled images per class) | accuracy: 92.28 -> 94.63 | Helped |
| Use task-oriented rather than task-agnostic VFM transfer | FastViT with OpenCLIP · Places365 (250 labeled images per class) | accuracy: 51.92 ± 0.26 -> 54.82 ± 0.04 | Helped |
| Use task-related rather than generic CC3M transfer data | Task-oriented VFM transfer to FastViT · HAM10K, EuroSAT, Places365, and ImageNet | accuracy: up to 5%; 7.3%; 2.3%; 10.9% | Helped |
| Use task-related rather than generic CC3M transfer data | Task-oriented VFM transfer to FastViT · ADE20K | mean IoU | Hurt |
| Add the final labeled-data finetuning step (ablation) | Task-oriented transfer to FastViT · ADE20K with CC3M transfer | performance: 2.19; 2.03 | Helped |
| Add the final labeled-data finetuning step (ablation) | Task-oriented transfer to FastViT · Classification tasks with task-related transfer data | performance | No clear effect |
| Finetune the VFM instead of linear-probing it (ablation) | Task-oriented transfer to FastViT with DINOv2 · HAM10K | accuracy: 74.60 -> 78.04 | Helped |
| Finetune the VFM instead of linear-probing it (ablation) | Task-oriented transfer to FastViT with OpenCLIP · EuroSAT | accuracy: 94.37 -> 94.63 | No clear effect |
| Use task-related rather than generic transfer data | Task-oriented transfer to FastViT on ADE20K · ADE20K segmentation | mean IoU | Mixed |
Surprising findings
- Finetuning the VFM helps less once the VFM is strong: linear-probed VFMs give target models close to those from finetuned VFMs (gap <0.5% for most datasets).
- The small target model sometimes beats its teacher VFM when a task-related transfer set is used.
Limitations
- Curating large transfer sets may be difficult for specialized domains (e.g., industrial automation) poorly covered by web data.
- Only labeled target data is used to adapt the VFM; large unlabeled data is not exploited for adaptation.
- Retrieval results depend on gallery coverage and de-contamination/de-duplication effort.
Experimental setup
Datasets: CC3M HAM10K EuroSAT Places365 ImageNet-1K ADE20K YFCC15M DataComp-1B
Benchmarks: HAM10K EuroSAT Places365 ImageNet ADE20K
Metrics: classification accuracy mean IoU A100 GPU hours
Baselines: task-agnostic VFM distillation CLIP-Pretrain IN-Pretrain DINO-Pretrain training from scratch
Builds on: knowledge distillation DINO CLIP DINOv2 DeepLabV3
Closest papers by meaning
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.89)
- Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model (ICLR 2024 · similarity 0.89)
- Accessing Vision Foundation Models via ImageNet-1K (ICLR 2025 · similarity 0.88)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.87)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.