Research map / Vision-language and multimodal / CLIP and zero-shot
AutoVP: An Automated Visual Prompting Framework and Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoVP, an end-to-end expandable framework that jointly tunes visual prompting design choices (input scale, prompt size, pre-trained classifier, output label mapping) and introduces a 12-dataset VP benchmark.
Problem
Visual prompting (VP) design choices (prompt size, source model, label mapping) have been studied separately, and VP methods are evaluated ad hoc on arbitrary datasets, making systematic comparison difficult.
Approach
An automated, modular framework that jointly searches input scaling, pixel-level frame prompts, pre-trained classifier (ResNet18, ResNeXt101-IG, Swin-T, CLIP) and four output label mappings (IterMap, FreqMap, SemanticMap, FullyMap) using Ray Tune grid search with ASHA early stopping, plus a unified 12-dataset benchmark with weight initialization for FullyMap on CLIP.
Key results
- With CLIP as the pre-trained model, AutoVP averages 80.6% over 12 datasets versus 78.5% for ILM-VP, 76.0% for CLIP-VP and 79.9% for linear probing.
- With ResNet18, AutoVP averages 70.02% versus 45.2% for ILM-VP and 69.3% for LP.
- With source-model selection unrestricted, AutoVP averages 81.02% versus 77.64% for LP of the chosen model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AutoVP's automatically tuned configuration with CLIP instead of ILM-VP | CLIP visual prompting · 12 downstream tasks (average) | accuracy (%): 78.5 -> 80.6 | Helped |
| use AutoVP's automatically tuned configuration with CLIP instead of CLIP-VP | CLIP visual prompting · 12 downstream tasks (average) | accuracy (%): 76.0 -> 80.6 | Helped |
| use AutoVP's automatically tuned configuration with CLIP instead of linear probing | CLIP · 12 downstream tasks (average) | accuracy (%): 79.9 -> 80.6 | Helped |
| automatically select a pre-trained model rather than fixing CLIP | AutoVP · 12 downstream tasks (average) | accuracy (%): 0.43% | Helped |
| use AutoVP rather than linear probing | CLIP visual prompting · CLEVR | accuracy (%): 68.0 -> 83.0 | Helped |
| add semantic weight initialization to FullyMap (ablation) | AutoVP with CLIP · CLIP visual prompting | accuracy | Helped |
| exclude the CLIP text encoder from the output mapping (ablation) | AutoVP with CLIP · 12 downstream tasks (average) | accuracy (%): 69.0% | Hurt |
| add visual prompts to the ResNet18 pipeline (ablation) | AutoVP with ResNet18 · 12 downstream tasks | accuracy | Mixed |
| add visual prompts to the CLIP pipeline (ablation) | AutoVP with CLIP · most downstream datasets | accuracy | Helped |
| use FullyMap instead of IterMap (ablation) | visual prompting · mapping analysis dataset | accuracy (%): 40.77% -> 69.96% | Helped |
| increase the number of source labels mapped to each target class (ablation) | AutoVP with vision models · downstream classification tasks | accuracy | Helped |
Surprising findings
- With ResNet18, visual prompts often hurt accuracy; gains are mostly attributed to FullyMap label mapping rather than the prompts.
- FullyMap with CLIP performs poorly during short tuning yet reaches higher accuracy after 200 epochs, so few-epoch tuning can select suboptimal settings.
Limitations
- Learning rate and weight decay are not included in the automated search.
- Scope is primarily image classification; segmentation and detection are only preliminary.
- Baseline numbers for CLIP-VP and ILM-VP are taken from prior papers for some datasets, with differing settings for others.
Experimental setup
Datasets: CIFAR10 CIFAR100 ISIC SVHN GTSRB Flowers102 DTD Food101 EuroSAT OxfordIIITPet UCF101 FMoW
Benchmarks: CIFAR10 CIFAR100 ISIC SVHN GTSRB Flowers102 DTD Food101 EuroSAT OxfordIIITPet UCF101 FMoW
Metrics: test accuracy average accuracy IoU pixel accuracy
Baselines: Linear Probing CLIP-TP CLIP-VP ILM-VP
Builds on: ILM-VP CLIP-VP Ray Tune ASHA
Closest papers by meaning
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.92)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.91)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.90)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.89)
- Selective Visual Prompting in Vision Mamba (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.