Research map / Architectures and efficiency / Pruning and efficient tuning
Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACAVP, a visual prompting method combining affine, color, and additive transformations with TrivialAugment to mitigate overfitting, achieving state-of-the-art VP accuracy on 12 image classification datasets.
Problem
Visual prompting (VP) for parameter-efficient adaptation of frozen vision models typically underperforms other adaptation methods; enlarging the prompt increases overfitting rather than expressivity.
Approach
Extend the additive-only VP transformation with a learnable affine transformation (creating prompt regions while keeping original image content) and a learnable color multiplier, and use TrivialAugment as a regularizer that substantially reduces VP overfitting and benefits existing VP methods too.
Key results
- ACAVP with CLIP ViT-B/32 reaches 83.18% average accuracy over 12 datasets, above EVP (78.86), AutoVP (81.54) and linear probing (82.59).
- TrivialAugment adds up to +16.26 points on Flowers for image-sized VP and +12.01 on Flowers for standard VP with CLIP ViT-B/32.
- ACAVP adds about 6% of CLIP ViT-B/32 inference time, and reaches 84.0% on CIFAR10-C versus 79.7% for AutoVP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACAVP instead of AutoVP | CLIP ViT-B/32 · 12-dataset average | accuracy: 83.18 vs 81.54 | Helped |
| Use ACAVP instead of linear probing | CLIP ViT-B/32 · 12-dataset average | accuracy: 83.18 vs 82.59 | Helped |
| Use ACAVP instead of AutoVP | CLIP ViT-B/32 · Flowers | accuracy: 6.67 percentage points | Helped |
| Use ACAVP instead of VP | CLIP ViT-B/32 · CIFAR10-C | accuracy: 7.33 percentage points | Helped |
| Add affine transformation to zero-shot input (ablation) | Visual prompting · CIFAR10 | accuracy: 88.93 -> 90.17 | Helped |
| Add affine transformation to zero-shot input (ablation) | Visual prompting · Flowers | accuracy: 61.63 -> 57.42 | Hurt |
| Add color transformation to zero-shot input (ablation) | Visual prompting · CIFAR10 and Flowers | accuracy: CIFAR10: 88.93 -> 92.14; Flowers: 61.63 -> 62.92 | Helped |
| Add affine and color transformations (ablation) | Visual prompting without additive prompting · Flowers | accuracy: 61.63 -> 60.58 | Hurt |
| Add color transformation to Resize + Additive (ablation) | EVP-style prompting · Flowers | accuracy: 79.24 -> 80.74 | Helped |
| Replace resize with affine transformation (ablation) | Additive visual prompting · Flowers | accuracy: 79.24 -> 88.70 | Helped |
| Add color transformation to Affine + Additive (ablation) | ACAVP · Flowers | accuracy: 88.70 -> 89.36 | Helped |
| Add TrivialAugment to image-sized VP (ablation) | Image-sized VP · CIFAR10 and Flowers | accuracy: CIFAR10: +2.04; Flowers: +16.26 | Helped |
Surprising findings
- Simply increasing the number of VP parameters via larger image-sized prompts lowered test accuracy on CIFAR10 (93.65 to 89.52), indicating overfitting rather than expressivity gains.
- With TrivialAugment, standard VP reaches accuracy comparable to EVP, suggesting EVP's resize operation mainly acted as an implicit regularizer.
Limitations
- Gains on some datasets are modest or negative relative to baselines (e.g., Food, DINOv2 on CIFAR10 and Flowers where AutoVP is best), and ACAVP is not best on all datasets.
- Results reported with CLIP primarily; ImageNet experiment is preliminary with only 10 epochs and modest gains.
- Inference overhead is higher than standard VP and EVP, and relative cost rises for lighter models (up to 0.49 of model time for DINOv2).
Experimental setup
Datasets: CIFAR10 CIFAR100 CLEVR DTD EuroSAT Flowers102 Food101 GTSRB OxfordPets SUN397 SVHN UCF101
Benchmarks: CIFAR10 CIFAR100 CLEVR DTD EuroSAT Flowers102 Food101 GTSRB OxfordPets SUN397 SVHN UCF101
Metrics: classification accuracy inference time relative to model
Baselines: ZS VP EVP AutoVP Linear probing Full fine-tuning Coordinator RandAugment IPMix
Builds on: EVP AutoVP TrivialAugment VP (padding-based visual reprogramming)
Closest papers by meaning
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.91)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.91)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.90)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.90)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.