Research map / Vision-language and multimodal / CLIP and zero-shot
Query Efficient Black-Box Visual Prompting with Subspace Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a query-efficient black-box visual prompting method that pre-trains a prompt-generating coordinator with BlackVIP, learns a low-dimensional subspace of its parameters via iterative zeroth-order optimization, then tunes within that subspace to reduce API queries to CLIP.
Problem
Black-box visual prompt learning needs many API queries to a pre-trained model because the prompt parameter space is high-dimensional, making zeroth-order optimization costly.
Approach
Reparameterize the visual prompt generator with a pre-trained initialization, learn a task-agnostic low-dimensional subspace by iteratively generating orthogonal projection columns with zeroth-order optimization, and augment it with random orthogonal directions so that black-box tuning converges with far fewer queries.
Key results
- Average API query count across 14 benchmarks is 2.6%, 4.3%, and 3.6% of VP, BAR, and BlackVIP respectively, per Table 2.
- Achieves 63.3% average accuracy with 0.15k trainable parameters, matching BlackVIP-ft (63.3%) and beating zero-shot CLIP on 13 of 14 benchmarks.
- On SVHN, subspace optimization reduces training epochs from 453 to 270 compared with optimizing the original space.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use subspace learning with pre-training for black-box visual prompting | CLIP ViT-B/16 visual prompting · 14 benchmarks (average) | API queries (1k): 246.9 -> 8.9 | Helped |
| Use subspace learning with pre-training instead of BlackVIP fine-tuning | CLIP ViT-B/16 visual prompting · 14 benchmarks (average) | API queries (1k): 12.6 -> 8.9 | Helped |
| Use subspace learning with pre-training instead of BlackVIP fine-tuning | CLIP ViT-B/16 visual prompting · 14 benchmarks (average) | test accuracy: 63.3 -> 63.3 | No clear effect |
| Add random orthogonal vectors to the projections (ablation) | Subspace learning visual prompting · SVHN | test accuracy: 41.1 -> 43.0 | Helped |
| Add random orthogonal vectors to the projections (ablation) | Subspace learning visual prompting · EuroSAT | test accuracy: 72.1 -> 72.3 | Helped |
| Learn the subspace instead of constructing it entirely from random orthogonal vectors (ablation) | Visual prompt tuning · CLEVR | test accuracy: 32.5 -> 32.9 | Helped |
| Add pre-training before low-dimensional subspace tuning (ablation) | Visual prompt tuning · SVHN | training epochs: 4600 -> 270 | Helped |
| Add pre-training before original-space tuning (ablation) | Visual prompt tuning · EuroSAT | training epochs: 2500 -> 173 | Helped |
| Optimize pre-trained parameters in a low-dimensional subspace instead of the original space (ablation) | Visual prompt tuning · SVHN | training epochs: 453 -> 270 | Helped |
| Optimize pre-trained parameters in a low-dimensional subspace instead of the original space (ablation) | Visual prompt tuning · EuroSAT | training epochs: 240 -> 173 | Helped |
Surprising findings
- Optimizing in the low-dimensional subspace without pre-training yields lower accuracy than optimizing in the original space, so the subspace mainly helps when built on pre-trained initialization.
Limitations
- Subspace learning stage itself requires iterative queries and is limited to a small subspace dimension (d=5) in experiments.
- Subspace-only optimization without pre-training underperforms full-space optimization, so the method depends on the pre-training stage.
- Evaluated only on CLIP ViT-B/16 as black-box model.
Experimental setup
Datasets: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD SVHN EuroSAT Resisc45 CLEVR
Benchmarks: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD SVHN EuroSAT Resisc45 CLEVR
Metrics: average test accuracy number of black-box API queries training epochs
Baselines: Zero-shot CLIP BAR VP (SPSA-GC) BlackVIP BlackVIP-ft
Builds on: BlackVIP Black-Box Tuning (BBT) BBTv2 Visual Prompting Coordinator (BlackVIP)
Closest papers by meaning
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.92)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.90)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.90)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.90)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.