Research map / Architectures and efficiency / Attention and transformers
Learning Expressive Prompting With Residuals for Vision Transformers
Rajshekhar Das, Yonatan Dukler, Avinash Ravichandran, Ashwin Swaminathan
This paper tried a common change: Add attention module
Add residual prompts at attention LayerNorm · FGVC · accuracy: +3.5% Improved
What 492 accepted papers report for this change
- 444 improved
- 20 no clear effect
- 4 mixed
- 24 hurt
- Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems ICML 2026 · add wavelet attention block · CKF step 64 · Rel L2: 1.09 -> 0.35Improved
- DA-DFGAS:Differentiable Federated Graph Neural Architecture Search with Distribution-Aware Attentive Aggregation AAAI 2026 · add attention mechanism to aggregation · Table 4 datasets · parameters: about 5-10% more parametersHurt
- Caracal: Causal Architecture via Spectral Mixing ICML 2026 · add sliding window attention layers to MHF stack · Avg accuracy · avg accuracy (%): 48.22 -> 49.01Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EXPRES instead of VPT-deep | ViT-B/16 adaptation · VTAB-1k natural mean | accuracy: 78.48 -> 79.7 | Improved |
| Use EXPRES instead of FT-all | ViT-B/16 adaptation · VTAB-1k specialized mean | accuracy: 83.36 -> 84.0 | Improved |
| Use EXPRES instead of VPT-deep | ViT-B/16 adaptation · VTAB-1k structured mean | accuracy: 54.98 -> 55.0 | No clear effect |
| Use EXPRES instead of VPT-deep | ViT-B/16 few-shot segmentation · PASCAL-5i 5-shot | mIoU: 33.4 -> 41.8; 56.2 -> 60.2; 49.8 -> 52.4; 47.7 -> 51.4 | Improved |
| Use EXPRES instead of VPT-deep | ViT-B/16 adaptation · Oxford Flowers | accuracy: 99.0 -> 99.0 | No clear effect |
| Use EXPRES instead of VPT-deep | ViT-B/16 adaptation · CUB | accuracy: 88.5 -> 88.3 | Hurt |
| Use EXPRES instead of VPT-deep | ViT-B/16 adaptation · Stanford Cars | accuracy: 83.6 -> 80.5 | Hurt |
| Use last-layer keys rather than queries or MLP outputs (ablation) | EXPRES for few-shot segmentation · PASCAL-5i fold 0 train-split, 5-shot | accuracy: 44.39 -> 51.61; 38.25 -> 51.61 | Improved |
| Allow prompt interactions with other tokens through more encoder layers (ablation) | EXPRES · FGVC and VTAB prompt propagation experiments | accuracy | Improved |
| Add residual prompts at attention LayerNorm (ablation) | EXPRES · FGVC | accuracy: +3.5% | Improved |
| Add residual prompts to query-key-value projections (ablation) | EXPRES · FGVC | accuracy: +2.8% | Improved |
| Prompt the attention block rather than the MLP block (ablation) | EXPRES · FGVC | accuracy: +1.6% | Improved |
Closest papers by meaning
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.90)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.88)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.88)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.88)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
