Research map / Adversarial robustness and security / Attacks on text and graphs
One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adversarial Prompt Tuning (APT), which learns robust soft text-prompt context vectors for frozen CLIP via adversarial training, improving adversarial robustness and accuracy with very few parameters and data.
Problem
Adversarial vulnerability of pre-trained vision-language models like CLIP, and how to adapt them for adversarial robustness parameter-efficiently without modifying model weights or images.
Approach
Shows that attack and defense strength depend on the text prompt used, then learns continuous prompt context vectors adversarially (with on-the-fly adversarial examples generated using the latest prompt) while keeping both CLIP encoders frozen.
Key results
- With 1 shot at eps=1/255, APT-UC improves accuracy and robustness over hand-engineered prompts by +6.1% and +3.0%.
- With full training data, APT-CSC beats HEP by +26.4% accuracy and +16.7% robustness at eps=4/255 (averaged over 11 datasets).
- In zero-shot transfer from ImageNet, APT improves average accuracy by 1.1% and robustness by 1.7% over TeCoA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune one shared soft-prompt context adversarially instead of using hand-engineered prompts | CLIP ViT-B/32 · 11-dataset average, all shots, ε=4/255 | accuracy: +21.9 points (54.9 vs 33.0) | Helped |
| Tune class-specific soft-prompt contexts adversarially instead of using hand-engineered prompts | CLIP ViT-B/32 · 11-dataset average, all shots, ε=4/255 | robustness: +16.7% | Helped |
| Use APT-UC rather than PAFT | CLIP ViT-B/32 · 11-dataset average, 1 shot, ε=4/255 | accuracy: 19.2 -> 33.1 | Helped |
| Use APT-UC rather than AVP | CLIP ViT-B/32 · 11-dataset average, 1 shot, ε=4/255 | robustness: 10.5 -> 11.4 | Helped |
| Use APT-UC rather than PAFT | CLIP ViT-B/32 · 11-dataset average, all shots, ε=4/255 | robustness: 27.5 -> 24.8 | Hurt |
| Use APT-UC rather than HEP | CLIP ViT-B/32 · ImageNet-Sketch OOD shift, 16 shots, ε=4/255 | robustness: 7.21 -> 8.04 | Helped |
| Use APT-UC rather than TeCoA + HEP | CLIP ViT-B/32 · Zero-shot average across 10 target datasets, ε=4/255 | accuracy: 32.3 -> 32.5 | Helped |
| Transfer the ImageNet-tuned APT-UC prompt to Flowers102 | CLIP ViT-B/32 · Flowers102 zero-shot, ε=4/255 | accuracy: 31.4 -> 29.1 | Hurt |
| Use unified rather than class-specific context with limited training data (ablation) | APT · 11-dataset average, 1 and 4 shots | accuracy: 51.3 vs 39.9 (1 shot); 58.2 vs 54.3 (4 shots), ε=1/255 | Helped |
| Use class-specific rather than unified context with more training data (ablation) | APT · 11-dataset average, all shots | accuracy: 73.5 vs 70.9, ε=1/255 | Helped |
| Generate training adversarial examples on-the-fly from the latest prompt rather than using a constant attack prompt (ablation) | APT · In-distribution evaluation | performance | Helped |
| Perturb both image and text during adversarial-example generation instead of perturbing the image only (ablation) | APT · In-distribution evaluation | performance | No clear effect |
Surprising findings
- The worst-case robustness of CLIP varies with the inference prompt; random character strings as prompts can give different robustness than hand-crafted ones.
- Prompt-only tuning fails to boost robustness for standardly pre-trained (non-robust) CLIP, so APT depends on robust pre-trained weights.
Limitations
- Learned context vectors are hard to interpret; nearest-word decodings are often irrelevant or uninterpretable.
- APT cannot boost robustness of standardly pre-trained, highly vulnerable VLMs; depends on robust pre-trained weights (TeCoA).
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Metrics: clean accuracy robust accuracy under PGD-100
Baselines: Hand-Engineered Prompts (HEP) Adversarial Visual Prompting (AVP) Partial Adversarial Fine-Tuning (PAFT) TeCoA
Builds on: CLIP CoOp TeCoA adversarial training (PGD) TRADES
Closest papers by meaning
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.93)
- TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.92)
- Few-Shot Adversarial Prompt Learning on Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.90)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.90)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.