Research map / Vision-language and multimodal / CLIP and zero-shot
Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes PEFT learning dynamics in few-shot VLM adaptation, finds a breakpoint between task-level feature learning and base-class specialization, and proposes 2SFS: LayerNorm tuning followed by a linear classifier initialized with base text embeddings.
Problem
Few-shot adaptation of vision-language models (CLIP) with few samples per category, including base-to-novel generalization, where PEFT methods overfit to base classes and degrade on novel categories.
Approach
PEFT learning dynamics split into a phase that improves both base and novel accuracy and a later phase that only specializes to base classes; 2SFS exploits this by tuning LayerNorm parameters for a fixed fraction of the budget, then freezing them and training a linear classifier initialized with base-class text embeddings, enabling category-level selective inference that skips the text encoder for
Key results
- Base-to-novel with ViT-B/16: 2SFS achieves 80.20% harmonic mean average across 11 datasets, above MMA (79.87%) and CLIP-LoRA (77.28%).
- All-to-all with ViT-L/14: 2SFS achieves 87.1% mean accuracy, versus 86.9% for CLIP-LoRA and 84.4% for MMA.
- Fixed hyperparameters (M=300, fraction 0.6) tuned only on ImageNet; 2SFS is more stable across settings than CLIP-LoRA and MMA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the two-stage recipe: first-stage LayerNorm tuning followed by classifier tuning | CLIP ViT-B/16 · Base-to-novel, average across 11 datasets | Harmonic mean: 79.87 -> 80.20 | Helped |
| Use 2SFS instead of CLIP-LoRA | CLIP · All-to-all, mean across 11 datasets, ViT-B/32 | Accuracy: 78.9 -> 79.2 | Helped |
| Use 2SFS instead of CLIP-LoRA | CLIP · All-to-all, mean across 11 datasets, ViT-L/14 | Accuracy: 86.9 -> 87.1 | Helped |
| Use 2SFS instead of MMA | CLIP ViT-B/16 · EuroSAT base-to-novel | Harmonic mean: 83.87 -> 79.29 | Hurt |
| Use 2SFS instead of MMA | CLIP ViT-B/16 · Stanford Cars base-to-novel | Harmonic mean: 75.70 -> 78.46 | Helped |
| Use 2SFS instead of MMA | CLIP ViT-B/16 · FGVC Aircraft base-to-novel | Harmonic mean: 38.33 -> 40.63 | Helped |
| Add classifier tuning after first-stage LayerNorm tuning (ablation) | 2SFS · Stanford Cars all-to-all | Accuracy: +10.6 | Helped |
| Add classifier tuning after first-stage LayerNorm tuning (ablation) | 2SFS · FGVC Aircraft all-to-all | Accuracy: +11.2 | Helped |
| Add classifier tuning after first-stage LayerNorm tuning (ablation) | 2SFS · Food-101 all-to-all | Accuracy: -1.4 | Hurt |
| Add classifier tuning after first-stage LayerNorm tuning (ablation) | 2SFS · Oxford Pets all-to-all | Accuracy: -0.4 | Hurt |
| Use batch size 1 during first-stage training (ablation) | 2SFS · EuroSAT base-to-novel | Harmonic mean: 79.29 -> 84.32% | Helped |
| Set the first-stage iteration fraction to 0.6 rather than values before or after it (ablation) | 2SFS · ImageNet base-to-novel validation | Harmonic mean | Mixed |
Surprising findings
- PEFT dynamics show a consistent breakpoint where further tuning improves base classes but degrades novel-class accuracy irrecoverably.
- EuroSAT is the worst case for 2SFS in base-to-novel, attributed to only 5 base categories causing early exit from the first stage.
Limitations
- Performance drops on Food-101 and Oxford Pets in the all-to-all setting relative to the first stage alone.
- Hyperparameters (M, fraction) are tuned on ImageNet and assumed to transfer; EuroSAT shows sensitivity to the number of base categories.
Experimental setup
Datasets: ImageNet Caltech101 SUN397 DTD FGVC Aircraft Oxford Pets Oxford Flowers 102 Stanford Cars Food-101 EuroSAT UCF-101
Benchmarks: ImageNet Caltech101 SUN397 DTD FGVC Aircraft Oxford Pets Oxford Flowers 102 Stanford Cars Food-101 EuroSAT UCF-101
Metrics: top-1 accuracy harmonic mean
Baselines: CLIP zero-shot CoOp CoCoOp MaPLe ProGrad KgCoOp CLIP-LoRA MMA TIP-Adapter-F CLIP-Adapter PLOT++ TaskRes
Builds on: CLIP CLIP-LoRA CoOp CoCoOp LayerNorm tuning
Closest papers by meaning
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.92)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.92)
- Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models (CVPR 2024 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.89)
- Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.