Research map / Vision-language and multimodal / CLIP and zero-shot
Token Coordinated Prompt Attention is Needed for Visual Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Coordinated Prompt Attention (TCPA), a plug-and-play module that gives CLS tokens and image tokens separate prompts and assigns prompts from pools to individual image tokens via cosine-similarity top-K masking, improving several visual prompting methods on HTA and VTAB.
Problem
Existing visual prompting methods for frozen ViTs use the same shared prompts for all tokens, ignoring the distinct roles of CLS vs. image tokens and the varying information in different image patches, which yields indistinguishable and biased features.
Approach
Disentangle prompts into CLS prompts and image prompts that interact only with their respective token types, and further split image prompts into a pool where each image token is matched (cosine similarity, top-K binary mask) to a subset of prompts; the mask is applied to the attention map so that one attention computation realizes token-specific prompt interactions.
Key results
- On HTA, VPT+TCPA reaches 87.0% average accuracy vs. 85.5% for VPT (+1.5), and VFPT+TCPA reaches 90.4% vs. 89.2%.
- DAMVP+TCPA averages 89.9% on HTA vs. 88.5% for DAMVP (+1.4).
- On VTAB natural/specialized/structured groups, VPT+TCPA improves by +1.2, +1.9, +1.2 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TCPA to VP | VP · HTA (10-dataset average) | accuracy: 83.4 -> 85.4 | Helped |
| Add TCPA to VPT | VPT · HTA (10-dataset average) | accuracy: 85.5 -> 87.0 | Helped |
| Add TCPA to DAMVP | DAMVP · HTA (10-dataset average) | accuracy: 88.5 -> 89.9 | Helped |
| Add TCPA to AutoVP | AutoVP · HTA (10-dataset average) | accuracy: 85.4 -> 87.2 | Helped |
| Add TCPA to VFPT | VFPT · HTA (10-dataset average) | accuracy: 89.2 -> 90.4 | Helped |
| Add TCPA to VPT | VPT · VTAB Natural, Specialized, and Structured | accuracy: 78.5 -> 79.7; 82.4 -> 84.3; 55.0 -> 56.2 | Helped |
| Add TCPA to VP | VP · VTAB Natural, Specialized, and Structured | accuracy: 77.3 -> 78.1; 80.1 -> 82.6; 53.8 -> 55.4 | Helped |
| Add TCPA to DAMVP | DAMVP · VTAB Natural, Specialized, and Structured | accuracy: 79.1 -> 80.4; 83.4 -> 85.5; 56.2 -> 57.1 | Helped |
| Add TCPA to AutoVP | AutoVP · VTAB Natural, Specialized, and Structured | accuracy: 78.4 -> 79.3; 83.1 -> 85.2; 55.8 -> 56.9 | Helped |
| Add R-TCPA, disentangling CLS and image prompts (ablation) | VPT · CUB, Dog, and GTSRB | accuracy: +0.8%-0.9% | Helped |
| Add T-TCPA, allowing image tokens to select prompts from a pool (ablation) | VPT+R-TCPA · CUB, Dog, and GTSRB | accuracy: +0.6%-1.1% | Helped |
| Use a very small or excessively large prompt pool instead of a moderate-sized pool (ablation) | TCPA · CUB | accuracy | Hurt |
Limitations
- Gains on some datasets are small (e.g., +0.1 to +0.6 points on some HTA entries), and the paper reports no variance or significance tests.
- Theoretical section relies on cited low-rank attention results and is loosely stated; the connection to the claimed benefit is mostly argued via visualizations.
Experimental setup
Datasets: DTD CUB-200-2011 NABirds Stanford Dogs Oxford Flowers Food101 CIFAR-100 CIFAR-10 GTSRB SVHN VTAB-1k
Benchmarks: HTA VTAB-1k
Metrics: accuracy
Baselines: Full fine-tuning Linear Partial MLP Sidetune Bias Adapter Former VP VPT DAMVP AutoVP
Builds on: VPT DAMVP AutoVP VFPT
Closest papers by meaning
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.88)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.87)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.87)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.86)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.