Research map / Vision-language and multimodal / CLIP and zero-shot
Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes decoupled visual prompts (DVP), which split CLIP class descriptions by LLM-generated causes or K-means clusters, train one visual prompt per partition, and combine them with a probabilistic reweighting matrix estimated by MLE.
Problem
Single visual prompts used in visual reprogramming (VR) of CLIP, trained on all class descriptions at once, may fail to capture diverse description aspects and may be biased toward less discriminative descriptions.
Approach
Decouple descriptions into partitions (by explicit causes via an LLM, or by unsupervised K-means clusters), train a separate visual prompt per partition, and integrate their outputs with a probabilistic reweighting matrix (PRM) learned by MLE counts; theoretically shown to lower empirical risk relative to standard VR.
Key results
- DVP-cse reaches 80.1% average accuracy across 11 datasets with ViT-B16 CLIP, vs 78.5% for AttrVR and 74.4% for VP.
- DVP-cls reaches 79.7% average with ViT-B16, and DVP-cse improves over AttrVR on RN50 (66.6 vs 64.6) and RN101 (69.6 vs 67.2).
- Ablation shows removing decoupling (single VP) drops DVP-cse average from 80.1 to 78.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DVP-cse instead of AttrVR | CLIP ViT-B16 visual reprogramming · 11 datasets (average) | accuracy: 78.5 -> 80.1 | Helped |
| Use DVP-cls instead of AttrVR | CLIP ViT-B16 visual reprogramming · 11 datasets (average) | accuracy: 78.5 -> 79.7 | Helped |
| Add cause-based prompt decoupling (ablation) | DVP-cse · 11 datasets (average) | accuracy: 78.7 -> 80.1 | Helped |
| Add cluster-based prompt decoupling (ablation) | DVP-cls · 11 datasets (average) | accuracy: 78.7 -> 79.7 | Helped |
| Add visual reprogramming (ablation) | DVP-cse · 11 datasets (average) | accuracy: 67.0 -> 80.1 | Helped |
| Add visual reprogramming (ablation) | DVP-cls · 11 datasets (average) | accuracy: 67.7 -> 79.7 | Helped |
| Add PRM (ablation) | DVP-cse · 11 datasets (average) | accuracy: 79.4 -> 80.1 | Helped |
| Add PRM (ablation) | DVP-cls · 11 datasets (average) | accuracy: 79.2 -> 79.7 | Helped |
| Use DVP-cse instead of AttrVR | CLIP ViT-B16 visual reprogramming · Food | accuracy: 85.9 -> 85.6 | Hurt |
| Increase the number of prompts without decoupling-and-reweighting | visual reprogramming · 11 downstream datasets | accuracy | Mixed |
Surprising findings
- DVP shows no improvement on the Food dataset, which the authors attribute to food being distinguished more by smell and taste than visual cues.
- Simply increasing the number of VPs without the decoupling-and-reweighting framework can reduce accuracy due to overfitting.
Limitations
- Gains are modest (about 1-2 points average over AttrVR) and no improvement on Food.
- Relies on LLM-generated causes or existing attribute descriptions, whose quality affects partitioning.
- Number of partitions v must be chosen as a hyperparameter per dataset.
Experimental setup
Datasets: FGVCAircraft Caltech101 StanfordCars DTD EuroSAT Flowers102 Food101 OxfordPets SUN397 UCF101 Resisc45
Benchmarks: FGVCAircraft Caltech101 StanfordCars DTD EuroSAT Flowers102 Food101 OxfordPets SUN397 UCF101 Resisc45
Metrics: accuracy mean accuracy HSIC
Baselines: VP AR AttrVR
Builds on: AttrVR visual prompt (VP) adversarial reprogramming (AR) CLIP
Closest papers by meaning
- Attribute-based Visual Reprogramming for Vision-Language Models (ICLR 2025 · similarity 0.90)
- Sample-specific Masks for Visual Reprogramming-based Prompting (ICML 2024 · similarity 0.90)
- Bayesian-guided Label Mapping for Visual Reprogramming (NeurIPS 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.