Research map / Adversarial robustness and security / Attacks on text and graphs
TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPT, a test-time defense that tunes bimodal CLIP prompts per test sample using multi-view entropy minimization and adversarial-clean embedding statistic alignment to improve zero-shot adversarial robustness.
Problem
Zero-shot adversarial robustness of vision-language models like CLIP: small adversarial perturbations degrade inference, and existing adversarial prompt tuning methods require task-specific training data and often hurt generalization.
Approach
An unsupervised, per-sample test-time prompt tuning defense: augmented views are filtered by low entropy, prompts are updated for one step by minimizing multi-view entropy plus aligning the per-layer mean/variance of embeddings with pre-computed adversarial and clean statistics from ImageNet, with no downstream labels needed.
Key results
- Under AutoAttack (eps=1/255) with ViT-B/16, TAPT improves zero-shot robustness over APT baselines by 32.6% (visual-only), 25.9% (V-L joint), and 36.6% (V-L independent) on average.
- TAPT with V-L independent prompts reaches about 53.8% average AutoAttack robust accuracy on ViT-B/16 (vanilla CLIP ~0.1%).
- TAPT adds about 0.095s (visual-only) to 0.166s (V-L joint) per image of inference overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply test-time visual prompt tuning instead of ImageNet-trained visual prompts | CLIP ViT-B/16 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 16.3 -> 48.9 (32.6↑) | Helped |
| Apply test-time V-L joint prompt tuning instead of ImageNet-trained V-L joint prompts | CLIP ViT-B/16 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 15.6 -> 41.5 (25.9↑) | Helped |
| Apply test-time V-L independent prompt tuning instead of ImageNet-trained V-L independent prompts | CLIP ViT-B/16 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 17.2 -> 53.8 (36.6↑) | Helped |
| Apply test-time visual prompt tuning instead of ImageNet-trained visual prompts | CLIP ViT-B/32 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 10.5 -> 44.6 (34.1↑) | Helped |
| Apply test-time V-L joint prompt tuning instead of ImageNet-trained V-L joint prompts | CLIP ViT-B/32 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 11.1 -> 42.5 (31.4↑) | Helped |
| Apply test-time V-L independent prompt tuning instead of ImageNet-trained V-L independent prompts | CLIP ViT-B/32 · 11-dataset average, AutoAttack, ε=1/255 | robust accuracy: 11.0 -> 49.0 (38.0↑) | Helped |
| Apply test-time visual prompt tuning instead of ImageNet-trained visual prompts | CLIP ViT-B/16 · 11-dataset average clean accuracy | accuracy: 54.5 -> 64.1 | Helped |
| Apply test-time V-L joint prompt tuning instead of ImageNet-trained V-L joint prompts | CLIP ViT-B/16 · 11-dataset average clean accuracy | accuracy: 51.0 -> 51.8 | Helped |
| Apply test-time V-L independent prompt tuning instead of ImageNet-trained V-L independent prompts | CLIP ViT-B/16 · 11-dataset average clean accuracy | accuracy: 51.0 -> 61.4 | Helped |
| Do not reset prompts during inference rather than resetting after every sample (ablation) | TAPT · ImageNet | robust accuracy: 49.92 -> 0.48 | Hurt |
| Do not reset prompts during inference rather than resetting after every sample (ablation) | TAPT · 10 zero-shot datasets average | robust accuracy: 47.85 -> 3.79 | Hurt |
| Increase the number of test-time adaptation steps from 0 to 1 (ablation) | TAPT · Most datasets and prompt designs, PGD-100 | robust accuracy | Helped |
Surprising findings
- Continuous prompt adaptation without resets (reset=all) collapses robust accuracy to about 0.48% on ImageNet, showing vulnerability to poisoning-style attacks.
Limitations
- Addresses only image-modality attacks; cross-modal attacks are not handled.
- Limited to image recognition tasks; not extended to VQA or reasoning.
- Per-sample test-time tuning adds inference overhead and requires multiple augmented views.
Experimental setup
Datasets: ImageNet Caltech101 DTD EuroSAT Pets Aircraft Food101 Flowers Cars SUN397 UCF101
Benchmarks: ImageNet Caltech101 DTD EuroSAT Pets Aircraft Food101 Flowers Cars SUN397 UCF101
Metrics: robust accuracy clean accuracy
Baselines: Vanilla CLIP APT-V APT-VLJ APT-VLI
Builds on: test-time prompt tuning (TPT) adversarial prompt tuning (APT) MEMO TeCoA
Closest papers by meaning
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.92)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.92)
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.