Research map / Vision-language and multimodal / CLIP and zero-shot
Black-Box Test-Time Prompt Tuning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes B2TPT, a gradient-free test-time prompt tuning method for black-box CLIP that optimizes low-dimensional intrinsic text and vision prompts with CMA-ES and uses a consistent-or-confident pseudo-labeling strategy.
Problem
Test-time prompt tuning of vision-language models is infeasible when models are served as black-box APIs (MaaS) because existing TPT methods rely on gradient backpropagation and labels are unavailable at test time.
Approach
Optimize joint text and vision prompts without gradients by evolving low-dimensional intrinsic prompts with CMA-ES, projecting them via fixed linear maps into prompt space, and generating pseudo-labels by consistency or confidence between zero-shot CLIP and the prompted model outputs.
Key results
- B2TPT averages 63.68% over ImageNet and its four variants versus 62.44% for gradient-based TPT and 59.11% for CLIP zero-shot.
- On the 10-dataset cross-dataset benchmark, B2TPT reaches 69.87% average accuracy versus 65.10% for TPT.
- Ablation on four datasets shows the CoC pseudo-labeling raises the average from 63.85% (CLIP pseudo-labels) to 64.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply B2TPT instead of TPT | CLIP ViT-B/16 · OOD benchmark (OOD Average) | accuracy: 60.81 -> 62.21 | Helped |
| apply B2TPT instead of DiffTPT | CLIP ViT-B/16 · Cross-dataset benchmark (Average) | accuracy: 65.47 -> 69.87 | Helped |
| apply B2TPT instead of C-TPT | CLIP ViT-B/16 · EuroSAT | accuracy: 48.70 -> 46.80 | Hurt |
| add CLIP zero-shot pseudo-labeling and prompt tuning (ablation) | B2TPT · Ablation study average | accuracy: 58.08 -> 63.85 | Helped |
| add the CoC pseudo-labeling strategy (ablation) | B2TPT · Ablation study average | accuracy: 63.85 -> 64.80 | Helped |
| add the CoC pseudo-labeling strategy (ablation) | B2TPT · Aircraft | accuracy: 30.89 -> 32.13 | Helped |
| add the CoC pseudo-labeling strategy (ablation) | B2TPT · ImageNet-A | accuracy: 54.48 -> 55.26 | Helped |
Surprising findings
- Black-box B2TPT beats gradient-based TPT methods such as DiffTPT and C-TPT on the reported benchmarks despite gradient-free optimization.
Limitations
- Evaluated only with CLIP ViT-B/16 as backbone; compatibility with other VLMs is claimed but not tested.
- Per-batch iterative CMA-ES optimization with a population of 30 across 4 iterations adds query cost at test time.
- Performance is weaker than DiffTPT on ImageNet-A and EuroSAT in some comparisons, and sensitive to hyperparameters such as intrinsic dimension and population size.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101
Metrics: accuracy
Baselines: CLIP zero-shot CoOp CoCoOp TPT DiffTPT C-TPT
Builds on: Test-time prompt tuning (TPT) CMA-ES BPT-VLM MaPLe CLIP
Closest papers by meaning
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.93)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.92)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.91)
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.