Research map / Label-efficient and robust learning / Recognition and long-tail
Scalable Transfer Learning with Expert Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-specific experts selected from the combined ImageNet21k and JFT pools instead of the generic baseline | ResNet50-v2 transfer on VTAB-1k · VTAB-1k All | accuracy: 69.8 -> 72.3 | Helped |
| Use kNN performance proxy rather than random expert selection | JFT full experts · VTAB-1k Natural | accuracy: 60.6 -> 79.7 | Helped |
| Use target labels in kNN expert selection rather than image-only domain prediction | JFT full experts · VTAB-1k All | accuracy: 69.1 -> 70.2 | Helped |
| Replace full experts with residual adapter experts (ablation) | ImageNet21k expert transfer · VTAB-1k All | accuracy: 71.4 -> 70.6 | Hurt |
| Replace full experts with residual adapter experts (ablation) | JFT expert transfer · VTAB-1k All | accuracy: 70.2 -> 71.1 | Helped |
| Combine experts trained on both upstream datasets instead of using JFT experts alone (ablation) | Expert transfer · VTAB-1k All | accuracy: 71.8 -> 72.3 | Helped |
| Use only the selected expert rather than the generic baseline | ImageNet21k transfer · VTAB-1k All | accuracy: 69.8 -> 71.4 | Helped |
| Use only the selected expert rather than the generic baseline | JFT transfer · VTAB-1k All | accuracy: 69.8 -> 70.2 | Helped |
| Use full-network downstream fine-tuning instead of fine-tuning only the adapters (ablation) | Selected adapter expert · Downstream transfer | accuracy | Helped |
| Select one expert per task using kNN rather than random routing | JFT adapter experts · VTAB-1k Natural | accuracy: 58.6 -> 79.0 | Helped |
| Select one expert per task using kNN rather than random routing | JFT full experts · VTAB-1k Specialized | accuracy: 81.2 -> 83.6 | Helped |
| Select experts using kNN rather than domain prediction | JFT full experts · VTAB-1k Structured | accuracy: 57.0 -> 55.3 | Hurt |
Closest papers by meaning
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.89)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.87)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.87)
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.