Research map / Vision-language and multimodal / CLIP and zero-shot
A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that state-of-the-art efficient transfer learning adapters for CLIP need per-task hyperparameter tuning on large labeled/test sets, and proposes CLAP, a class-adaptive constrained linear probe whose penalty weights are set via an augmented Lagrangian method using only the support set.
Problem
Few-shot adaptation of large vision-language models (CLIP) with efficient transfer learning adapters relies on hyperparameters tuned on large validation or test sets, which is unrealistic in real-world settings; these methods also underperform zero-shot under distribution shift.
Approach
Revisits linear probing initialized with CLIP zero-shot prototypes (ZS-LP) as a strong baseline, then adds a constraint keeping prototypes near zero-shot anchors, with class-wise penalty weights learned by a one-step augmented Lagrangian (PHR) update computed from support-set zero-shot softmax scores, avoiding any validation set.
Key results
- CLAP with ResNet-50 reaches 62.79% (K=1) and 74.57% (K=16) average accuracy over 11 datasets, outperforming TaskRes(e) (61.44%, 74.42%) and CLIP-Adapter (58.43%, 73.35%) under a validation-free protocol.
- On ImageNet domain generalization with ViT-B/16, CLAP reaches 73.38 source and 60.04 target average, while TIP-Adapter(f) and TIP-Adapter drop below zero-shot on the source domain (51.71 and 53.81).
- In the 8-shot fine-tuning vs ETL comparison with ViT-B/16, CLAP achieves 59.21 target average versus 58.28 for plain fine-tuning, while using a fraction of the trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLAP's class-adaptive constraint to few-shot adaptation | ZS-LP on 11 datasets, CLIP ResNet-50 · 11-dataset average, K=1 to K=16 | accuracy: 61.28 -> 62.79; 64.88 -> 66.07; 67.98 -> 69.13; 71.43 -> 72.08; 74.37 -> 74.57 | Helped |
| Adapt with CLAP instead of zero-shot inference | CLIP ResNet-50 · ImageNet target average | accuracy: 40.61 -> 42.91 | Helped |
| Adapt with CLAP instead of zero-shot inference | CLIP ViT-B/16 · ImageNet target average | accuracy: 57.17 -> 60.04 | Helped |
| Use CLAP rather than FLYP with the stated 8-shot comparison protocol | CLIP ViT-B/16 · ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R | accuracy: 64.59 -> 64.06; 49.50 -> 47.66; 51.32 -> 48.40; 78.52 -> 76.70 | Mixed |
| Add data augmentation (ablation) | ZS-LP on 11 datasets · K=1, K=2, and K=4 | accuracy: 57.72(−3.5)↓; 61.94(−2.9)↓; 65.41(−2.5)↓ | Helped |
| Add CLIP temperature scaling (ablation) | ZS-LP on 11 datasets · K=1, K=2, and K=4 | accuracy: 58.33(−2.9)↓; 59.85(−5.0)↓; 59.91(−8.0)↓ | Helped |
| Add L2 normalization of class prototypes (ablation) | ZS-LP on 11 datasets · K=1, K=2, and K=4 | accuracy: 48.67(−12.6)↓; 55.29(−9.6)↓; 61.16(−6.8)↓ | Helped |
| Initialize class prototypes with zero-shot weights rather than random weights (ablation) | Linear Probing on 11 datasets · K=1, K=2, and K=4 | accuracy: 30.42(−30.8)↓; 41.86(−23.0)↓; 51.69(−16.2)↓ | Helped |
Surprising findings
- Many SoTA adapters (CLIP-Adapter, TIP-Adapter) fall well below a simple zero-shot-initialized linear probe and, under distribution shift, even below zero-shot inference.
- Hyperparameters tuned for one dataset transfer poorly to others, with degradations up to 21% relative to ZS-LP, suggesting prior results relied on test-set tuning.
Limitations
- Benefits of CLAP diminish as the number of shots increases, so other strategies may be preferable with many samples.
- Only one outer ALM iteration is used to avoid overfitting the support set, a heuristic choice.
- Some compared results (prompt learning, fine-tuning) are taken from prior papers rather than rerun under the same protocol.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: accuracy
Baselines: CLIP-Adapter TIP-Adapter TIP-Adapter(f) TaskRes(e) CrossModal-LP CoOp ProGrad PLOT LP-FT WiSE-FT FLYP
Builds on: CLIP CLIP-Adapter TIP-Adapter TaskRes Augmented Lagrangian method
Closest papers by meaning
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.92)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.91)
- Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models (CVPR 2024 · similarity 0.90)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.90)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.