Research map / Adversarial robustness and security / Attacks on text and graphs
Adversarial Prompt Tuning for Vision-Language Models
Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, Jitao Sang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune learnable text prompts against an adversarial embedding bank (AdvPT) instead of hand-crafted prompts | CLIP ViT-B/16 · ImageNet (PGD-40) | robust accuracy (%): 6.6 -> 19.9 | Improved |
| apply AdvPT against AutoAttack | CLIP ViT-B/16 · Flowers (AutoAttack) | accuracy (%): 0.0 -> 23.3 | Improved |
| combine AdvPT with Super-resolution input denoising | Super-resolution defense, CLIP ViT-B/16 · Flowers (PGD-40) | robust accuracy (%): 13.8 -> 60.4 | Improved |
| combine AdvPT with DiffPure | DiffPure, CLIP ViT-L/14 · Pets (PGD-40) | robust accuracy (%): 90.6 -> 90.3 | No clear effect |
| combine AdvPT with DiffPure | DiffPure, CLIP ViT-B/16 · Flowers (PGD-40) | robust accuracy (%): 59.4 -> 81.9 | Improved |
| train prompts with adversarial objective (AdvPT) instead of clean CoOp objective (ablation) | prompt tuning on CLIP · multiple datasets | robustness vs generalization | Mixed |
| use AdvPT instead of adversarial training (AT) | CLIP on Pets · Pets | training time: at least 100x more efficient | Improved |
| linear probing on downstream data instead of adversarial prompt tuning (ablation) | CLIP · Flowers, Pets and others (PGD-40) | robust accuracy (%): 6.4 -> 4.8 | Hurt |
| vary number of learnable vectors M from 1 to 50 (ablation) | AdvPT · Pets | robust accuracy | No clear effect |
| scale model from ViT-B/16 to ViT-L/14 (ablation) | AdvPT · generalization gap vs CoOp | clean accuracy | Improved |
| train prompts on ImageNet and test on ImageNet variants | AdvPT · ImageNetV2, Sketch, A, R | robust accuracy | Improved |
Closest papers by meaning
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.94)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.92)
- TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.90)
- Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning (NeurIPS 2025 · similarity 0.89)
- Few-Shot Adversarial Prompt Learning on Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
