Research map / Vision-language and multimodal / CLIP and zero-shot
Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skip Tuning, which makes full fine-tuning of CLIP efficient by caching shallow-layer features (LSkip) and sampling only a subset of class tokens per image (CSkip), with no extra context vectors or adapters.
Problem
Prompt tuning of pre-trained VLMs freezes weights, giving little memory/time savings and lower accuracy than full fine-tuning; the goal is an FT-level method that is both effective and memory/time efficient.
Approach
Shrink the feature-gradient propagation flows of full fine-tuning in both length (cache outputs of the first ω layers of CLIP's encoders and fine-tune only the remaining layers) and width (per image, compute the ITM loss over a sampled subset of class tokens using an exponential image-conditioned sampler).
Key results
- Base-to-new: Skip Tuning reaches 81.11% H accuracy averaged over 11 datasets, 1.14 points above PromptSRC, at about 7.2x less time and 5.1x less memory.
- Cross-dataset: Skip Tuning achieves 67.00% average accuracy on 10 target datasets with about 44x time and 21.5x memory efficiency versus PromptSRC.
- Against adapter methods, Skip Tuning gets 81.11 H vs 77.52 for LoRA, with about 3.8x time and 3.9x memory efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Skip Tuning (LSkip + CSkip) instead of PromptSRC | CLIP ViT-B/16 · Base-to-new generalization, 11 datasets (Avg) | H: 1.14% | Helped |
| Apply Skip Tuning instead of PromptSRC | CLIP ViT-B/16 · Cross-dataset generalization, 10 target datasets | ACC: 1.19% | Helped |
| Apply Skip Tuning instead of PromptSRC | CLIP ViT-B/16 · Domain generalization, 4 ImageNet variants | ACC: 0.55% | Helped |
| Add LSkip to full fine-tuning (ablation) | CLIP ViT-B/16 · Base-to-new generalization, 11 datasets (Avg) | H: 79.27 -> 81.00 | Helped |
| Add CSkip to full fine-tuning (ablation) | CLIP ViT-B/16 · Base-to-new generalization, 11 datasets (Avg) | H: 79.27 -> 79.29 | No clear effect |
| Combine LSkip and CSkip (ablation) | CLIP ViT-B/16 · Base-to-new generalization, 11 datasets (Avg) | H: 79.27 -> 81.11 | Helped |
| Increase the number of skipped layers ω from 6 toward 10 (ablation) | Skip Tuning LSkip · 11-dataset average testing results | H | Hurt |
| Increase CSkip sampling ratio r above 0.7 (ablation) | Skip Tuning CSkip · 11-dataset average testing results | H | Hurt |
| Vary CSkip sampling ratio r from 0.4 to 1.0 (ablation) | Skip Tuning CSkip · 11-dataset average testing results | H | No clear effect |
| Increase CSkip decay coefficient λ from 0.7 to 1.0 (ablation) | Skip Tuning CSkip · 11-dataset average testing results | H | Hurt |
| Vary CSkip decay coefficient λ from 0.1 to 0.7 (ablation) | Skip Tuning CSkip · 11-dataset average testing results | H | No clear effect |
Surprising findings
- Ignoring most class tokens per image (CSkip with r=0.5) improves generalization rather than hurting it, which the authors attribute to filtering distracting text features.
- Freezing VLM weights during prompt tuning gives only 6.3% memory and 15.8% time savings over full fine-tuning while lowering accuracy.
Limitations
- Hyperparameters (ω, r, λ) were tuned on the same benchmarks reported, and the method is evaluated only on CLIP ViT-B/16.
- Efficiency comparisons depend on the authors' reproduced baselines and a single NVIDIA V100 GPU.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean accuracy training time (s) memory (M)
Baselines: CoOp CoCoOp ProGrad KgCoOp MaPLe PromptSRC TCP DePT CoPrompt CLIP-adapter LoRA
Builds on: CLIP CoOp DePT
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.92)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.90)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.90)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.