Research map / Vision-language and multimodal / CLIP and zero-shot
Adaptive Parameter Selection for Tuning Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-AST, which selects the top-K sub-layer parameters per transformer layer using AdamW second-moment estimates as importance scores, then fine-tunes only those without adding extra parameters.
Problem
PEFT methods for CLIP rely on manually chosen fine-tuning positions (prompts or adapters), and automatically selecting which existing parameters to tune in VLMs is underexplored.
Approach
Uses the square root of AdamW's second-moment estimates from a short transformer fine-tuning stage as per-sub-layer importance scores, then fine-tunes only the top-K sub-layers in each transformer layer while freezing the rest, with SCL regularization and no extra parameters.
Key results
- With 16-shot learning, CLIP-AST surpasses GraphAdapter and PromptSRC by 3.56% and 2.20% average accuracy on 11 datasets (abstract).
- On ImageNet 16-shot, CLIP-AST reaches 73.91% versus 73.40% for GraphAdapter and 73.17% for PromptSRC (Table 1).
- In the out-of-distribution setting, CLIP-AST achieves 62.87% average accuracy, versus 61.72% for PromptSRC (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive selective tuning instead of GraphAdapter | CLIP ViT-B/16 · ImageNet, 16-shot | accuracy: 73.40 -> 73.91 | Helped |
| Use CLIP-AST instead of PromptSRC | CLIP ViT-B/16 · Base-to-novel average | HM: 79.97 -> 81.06 | Helped |
| Use CLIP-AST instead of MMA | CLIP ViT-B/16 · Food101, base-to-novel | HM: 91.38 -> 90.84 | Hurt |
| Use CLIP-AST instead of PromptSRC | CLIP ViT-B/16 · ImageNet-Sketch | accuracy: 49.55 -> 48.37 | Hurt |
| Use CLIP-AST instead of PromptSRC | CLIP ViT-B/16 · SUN397, base-to-novel | HM: 80.52 -> 80.51 | Hurt |
| Fine-tune the image encoder only rather than freezing both encoders (ablation) | CLIP-AST · Few-shot, 16-shot, average across 11 datasets | accuracy: 65.21 -> 84.04 | Helped |
| Fine-tune the text encoder only rather than freezing both encoders (ablation) | CLIP-AST · Few-shot, 16-shot, average across 11 datasets | accuracy: 65.21 -> 81.95 | Helped |
| Use fixed bias-only selection instead of adaptive layer-granularity selection (ablation) | CLIP-AST · Few-shot, 16-shot, average across 11 datasets | accuracy: 85.05 -> 79.96 | Hurt |
| Select top K sub-layers globally rather than within each transformer layer (ablation) | CLIP-AST · Few-shot, 16-shot, average across 11 datasets | accuracy: 84.25 -> 85.05 | Hurt |
| Vary transformer fine-tuning epochs for importance scores (ablation) | CLIP-AST · Few-shot learning, average across 11 datasets | accuracy | No clear effect |
| Vary the number of selected sub-layers K (ablation) | CLIP-AST · Few-shot learning, average across 11 datasets | accuracy | Mixed |
Surprising findings
- BitFit-style fixed bias-only selection performed poorly (79.96% vs 85.05% for adaptive layer-granularity selection), suggesting prior-knowledge selection is suboptimal.
Limitations
- Top-K sub-layer count K (chosen as 6) is a hyperparameter selected by ablation rather than fully adaptive.
- Requires a preliminary transformer fine-tuning stage to obtain importance scores, adding a two-stage pipeline.
- Evaluated only on ViT-B/16 CLIP and image classification benchmarks.
Experimental setup
Datasets: ImageNet Caltech101 DTD EuroSAT FGVC Aircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNet-Sketch
Benchmarks: ImageNet Caltech101 DTD EuroSAT FGVC Aircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNet-Sketch
Metrics: top-1 accuracy base accuracy novel accuracy harmonic mean average accuracy training time FPS
Baselines: CLIP CoOp CoCoOp Tip-Adapter PromptSRC GraphAdapter MaPLe MMA
Builds on: CLIP AdamW PromptSRC BitFit BSR
Closest papers by meaning
- FATE: Feature-Adapted Parameter Tuning for Vision-Language Models (AAAI 2025 · similarity 0.92)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.91)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.90)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.90)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (ICML 2024 · similarity 0.90)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.