Research map / Adversarial robustness and security / Attacks on text and graphs
On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AOM, a training-free and truly zero-shot defense that moves adversarial CLIP features toward a Gaussian-noise-averaged anchor via one-step linear interpolation, improving zero-shot robust accuracy across 16 datasets.
Problem
Zero-shot adversarial robustness of vision-language models like CLIP, where existing defenses rely on adversarial fine-tuning on an extra dataset (e.g., TinyImageNet), which risks overfitting and violates the true zero-shot setting.
Approach
Adding Gaussian noise to adversarial inputs weakens the attack; averaging the features of several noisy copies gives an anchor, and the adversarial feature is linearly interpolated toward (and slightly beyond) that anchor in one step, with no training or clean data.
Key results
- Average zero-shot robust accuracy under PGD-10 (eps=1/255) of 41.72% vs 31.95% for PMG-AFT (+9.77%) and 17.38% for CLIP.
- Average clean zero-shot accuracy of 62.24%, 6.53% above PMG-AFT, though 2.31% below original CLIP (64.55%).
- On 11 datasets under PGD-100 (eps=4/255), AOM reaches 38.1 robust accuracy vs 27.0 for APT, while clean accuracy is 57.4 vs 59.4 for APT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AOM instead of PMG-AFT | CLIP ViT-B/32 · 16 datasets, PGD-10 | average robust accuracy: 31.95 -> 41.72; 9.77% | Helped |
| Apply AOM instead of PMG-AFT | CLIP ViT-B/32 · 16 datasets, AutoAttack | average robust accuracy: 17.91 -> 44.79; 26.88% | Helped |
| Apply AOM instead of PMG-AFT | CLIP ViT-B/32 · 16 datasets (clean) | average accuracy: 55.71 -> 62.24; 6.53% | Helped |
| Apply AOM instead of APT | CLIP ViT-B/32 · 11 datasets, PGD-100, ϵ = 4/255 | average robust accuracy: 27.05 -> 38.11; 11.1% | Helped |
| Add Gaussian noise to adversarial samples (ablation) | CLIP · 16 datasets, PGD10 | robust accuracy | Helped |
| Increase Gaussian noise level from σ = 0.08 to σ = 0.18 (ablation) | AOM · TinyImageNet, PGD-100 | robust accuracy | Helped |
| Increase Gaussian noise level from σ = 0.08 to σ = 0.18 (ablation) | AOM · TinyImageNet (clean) | accuracy | Hurt |
| Increase Gaussian noise level to σ = 0.26 or σ = 0.38 (ablation) | AOM · TinyImageNet, PGD-100 | robust accuracy | Hurt |
| Increase interpolation coefficient α from 0 toward 1.2 (ablation) | AOM · ImageNet, PGD-100 | robust accuracy | Helped |
| Increase interpolation coefficient α (ablation) | AOM · ImageNet (clean) | accuracy | Hurt |
Surprising findings
- Simply adding small Gaussian noise to adversarial examples substantially improves CLIP's zero-shot robust accuracy.
- The best step size alpha was 1.2, extending the movement beyond the anchor rather than stopping at it.
Limitations
- Clean accuracy drops relative to the original CLIP (about 2.3 points average), since clean inputs also undergo the movement.
- Robust accuracy is sensitive to sigma and alpha, which were tuned on the evaluation datasets, and only CLIP ViT-B/32 was tested.
- Evaluation relies on baseline numbers copied from prior papers, and some attack settings differ across comparisons.
Experimental setup
Datasets: TinyImageNet CIFAR10 CIFAR100 STL10 Caltech256 ImageNet Caltech101 Flowers102 OxfordPets StanfordCars FGVC-Aircraft SUN397
Benchmarks: CIFAR10 CIFAR100 STL10 Caltech256 ImageNet Caltech101 Flowers102 OxfordPets StanfordCars FGVC-Aircraft SUN397 PCAM
Metrics: top-1 clean accuracy top-1 robust accuracy
Baselines: FT-TeCoA PMG-AFT HEP AVP PAFT APT FT-Standard
Builds on: CLIP random smoothing TeCoA PMG-AFT PGD
Closest papers by meaning
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.93)
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness (CVPR 2024 · similarity 0.91)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.90)
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.90)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.