Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning

ICLR 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Fine-tuning strategies for few-shot · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Provides a theoretical analysis of multitask finetuning of foundation models before few-shot target adaptation, introduces diversity/consistency quantities, and a greedy coverage-based task selection algorithm validated on Meta-Dataset and few-shot image benchmarks.

Problem

Few-shot adaptation of pretrained foundation models to new tasks with limited labels lacks theoretical understanding, particularly when finetuning on auxiliary multitask data before target adaptation.

Approach

Derives error bounds for multitask finetuning in terms of a diversity parameter (coverage of target by auxiliary task solutions) and consistency parameter (similarity between auxiliary and target optimal solutions), and uses cosine similarity plus a Gaussian coverage score to greedily select finetuning tasks.

Key results

• On DomainNet 1-shot with DINO v2 ViT-S, multitask finetuning reaches 61.57% vs 52.28% direct adaptation (a gain of about 9.3 points).
• On Meta-Dataset Omniglot, the selection algorithm gives gains over random selection of 8.9%, 19.7%, and 2.4% with CLIP, DINOv2, and MoCo v3 respectively.
• Multitask finetuning on supervised ViT-B yields 96.91% (1-shot) on miniImageNet vs 94.06% for direct adaptation.

Measured effects reported in the paper

Surprising findings

• Number of shots per finetuning task matters little; total sample size M*m determines performance, contrary to the belief that finetuning must mimic the exact few-shot setting.
• Multitask finetuning gains are larger for supervised-pretrained models than for DINO v2 models, suggesting a larger domain gap for supervised pretraining.

Limitations

• Estimating diversity and consistency parameters on real-world datasets is time-consuming and no efficient estimator is known.
• Experimental results may not beat state-of-the-art, since the focus is on theory rather than SOTA performance.
• Theory relies on linear heads, linear data models and regularity assumptions.

Experimental setup

Datasets: miniImageNet tieredImageNet DomainNet Meta-Dataset

Benchmarks: miniImageNet tieredImageNet DomainNet Meta-Dataset

Metrics: classification accuracy 95% confidence intervals

Baselines: Direct Adaptation Standard FT Random task selection No Consistency selection No Diversity selection

Builds on: Prototypical Networks contrastive learning theory (Haochen et al.) multitask representation learning diversity (Tripuraneni et al.) MAML-style meta-learning

Closest papers by meaning

• Overcoming Generic Knowledge Loss with Selective Parameter Update (CVPR 2024 · similarity 0.88)
• Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.86)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.86)
• Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models (ICML 2024 · similarity 0.86)
• Provable Meta-Learning with Low-Rank Adaptations (NeurIPS 2025 · similarity 0.86)
• A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints (ICML 2025 · similarity 0.86)