Research map / Large language models / Language, safety and interpretability
ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZIP, a zeroth-order prompt-tuning method for black-box vision-language models that uses low-rank diagonal reparameterization, feature sharing, and intrinsic-dimensional gradient clipping to reach higher accuracy with far fewer queries.
Problem
Black-box prompt tuning (BBPT) of vision-language models requires excessive API queries, and naive zeroth-order optimization degrades as prompt dimensionality grows, which is impractical under limited query budgets.
Approach
Reduce the trainable prompt dimension via Fastfood random projection plus a low-rank U diag(s) V^T factorization with feature sharing, and clip zeroth-order gradient estimates at the threshold sqrt(delta) (delta = intrinsic parameter count), removing the need to tune a clipping threshold.
Key results
- Few-shot (16-shot) average accuracy of 63.4% across 13 tasks vs 57.3% manual prompt, 56.6% BlackVIP, 55.7% BAR, with 0.4k parameters and 5,000 queries.
- Base-to-new harmonic mean average 67.9% vs 63.5% for BlackVIP (best baseline), and cross-dataset/OOD averages of 54.8% and 56.9%.
- Claims over 48% query-efficiency improvement over the second-best BBPT method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZIP instead of BlackVIP | CLIP black-box prompt tuning · Few-shot, 16-shot average | accuracy: 56.6 -> 63.4 | Helped |
| Use ZIP instead of the strongest baseline | CLIP black-box prompt tuning · Base-to-new, base average | accuracy: 65.4 -> 71.6 | Helped |
| Use ZIP instead of the strongest baseline | CLIP black-box prompt tuning · Base-to-new, new average | accuracy: 64.5 -> 65.2 | Helped |
| Use ZIP instead of the strongest baseline | CLIP black-box prompt tuning · Base-to-new, harmonic mean average | accuracy: 62.9 -> 67.9 | Helped |
| Use ZIP instead of BlackVIP | CLIP prompt tuning trained on ImageNet · Cross-dataset transfer average | accuracy: 54.1 -> 54.8 | Helped |
| Use ZIP instead of BlackVIP | CLIP prompt tuning trained on ImageNet · OOD average | accuracy: 54.9 -> 56.9 | Helped |
| Increase the API query budget from 5,000 to 20,000 | ZIP on ImageNet · Few-shot, 16-shot | accuracy: 66.2% -> 67.2% | Helped |
| Add feature sharing (ablation) | ZIP · Few-shot average | accuracy: 60.2% to 62.4% | Helped |
| Add feature sharing (ablation) | ZIP · Base-to-new, base average | accuracy: 69.3 -> 71.6 | Helped |
| Add feature sharing (ablation) | ZIP · Base-to-new, new average | accuracy: 63.6 -> 65.0 | Helped |
| Add feature sharing (ablation) | ZIP · Base-to-new, harmonic mean average | accuracy: 66.0 -> 68.2 | Helped |
| Add feature sharing (ablation) | ZIP · Cross-dataset transfer average | accuracy: 54.0 -> 54.2 | Helped |
Surprising findings
- Naive zeroth-order prompt tuning degrades as the number of context tokens increases, unlike first-order prompt tuning.
Limitations
- Gains over baselines are mixed in some settings, e.g. SigLIP results where ZIP is not best on all datasets and cross-dataset transfer is slightly behind BlackVIP on some targets.
- Convergence analysis relies on assumptions (e.g. Bernoulli perturbations, c to 0 limit) that are idealized relative to practical training.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets Flowers102 Food101 FGVCAircraft SUN397 Resisc45 DTD SVHN EuroSAT CLEVR
Benchmarks: ImageNet Caltech101 OxfordPets Flowers102 Food101 FGVCAircraft SUN397 Resisc45 DTD SVHN EuroSAT CLEVR
Metrics: accuracy harmonic mean number of queries to target accuracy
Baselines: Manual Prompt BAR BlackVIP BPTVLM ZO-SGD first-order SGD LoRA
Builds on: BlackVIP BAR SPSA ZO-SGD intrinsic dimension
Closest papers by meaning
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.93)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.91)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.91)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.