Research map / Adversarial robustness and security / Attacks on text and graphs
Language-Driven Anchors for Zero-Shot Adversarial Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAAT, an anchor-based adversarial training method that uses fixed CLIP text-embedding anchors (after an expansion algorithm that reduces their high cosine similarity) plus an alignment cross-entropy loss and smoothness loss, to give image classifiers zero-shot adversarial robustness.
Problem
Adversarial robustness has been studied mainly in the fully supervised setting; extending adversarial training to zero-shot recognition of novel categories is hard, and prior CLIP-based attempts (TeCoA) only work at tiny perturbation budgets.
Approach
Use CLIP text encoder embeddings of class names as fixed normalized anchors for adversarial training. CLIP anchors have very high mutual cosine similarity, which hampers AT, so an expansion algorithm in spherical coordinates spreads them over the hemisphere while preserving semantic neighborhoods; an alignment cross-entropy loss and a benign-adversarial smoothness loss complete the objective.
Key results
- In the 5-way zero-shot setting on CIFAR-FS at eps=8/255, LAAT reaches 55.60 clean and 40.12 PGD-20 robust accuracy, above GR's few-shot robust accuracy (about 38.03 PGD).
- Models trained on ImageNet-1K with LAAT (ViT-B/16) reach 34.97% robust accuracy on aPY and 26.21% on AwA2 under PGD-20 at eps=8/255, versus at most about 6.45% for TeCoA.
- Expansion reduces the average CoS between CIFAR100 CLIP anchors from 0.746-0.779 to about 0.2 (e.g., 0.199 for ViT-B/32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LAAT rather than GR | 5-way zero-shot CIFAR-FS · CIFAR-FS | PGD accuracy: 38.03 ± 0.46 -> 40.12 ± 0.44 | Helped |
| Use LAAT rather than GR | 5-way zero-shot MiniImageNet · MiniImageNet | PGD accuracy: 27.57 ± 0.38 -> 29.27 ± 0.31 | Helped |
| Train on ImageNet-1K with LAAT rather than TeCoA | ViT-B/16 · CIFAR100 | PGD-20 adversarial accuracy: 0.20 -> 7.32 | Helped |
| Train on ImageNet-1K with LAAT rather than TeCoA | ViT-B/16 · aPY | PGD-20 adversarial accuracy: 6.45 -> 34.97 | Helped |
| Train on ImageNet-1K with LAAT rather than TeCoA | ViT-B/16 · AwA2 | PGD-20 adversarial accuracy: 1.95 -> 26.21 | Helped |
| Add expansion to the anchors (ablation) | Anchor-based adversarial training on CIFAR-FS · CIFAR-FS | Robust accuracy | Helped |
| Add A-CE loss (ablation) | Expanded-anchor adversarial training on CIFAR-FS · CIFAR-FS | Robust accuracy: 32.71 -> 37.69 | Helped |
| Add smoothness loss (ablation) | EXP + A-CE on CIFAR-FS · CIFAR-FS | Robust accuracy: 37.69 -> 40.12 | Helped |
| Replace smoothness loss with TRADES (1/λ = 1) (ablation) | EXP + A-CE on CIFAR-FS · CIFAR-FS | Robust accuracy: 40.12 -> 19.22 | Hurt |
| Replace smoothness loss with TRADES (1/λ = 6) (ablation) | EXP + A-CE on CIFAR-FS · CIFAR-FS | Robust accuracy: 40.12 -> 25.02 | Hurt |
| Set smoothness-loss α to 1.0 rather than 3.0 (ablation) | EXP + A-CE + SM on CIFAR-FS · CIFAR-FS | Robust accuracy: 40.12 -> 39.95 | No clear effect |
| Set smoothness-loss α to 6.0 rather than 3.0 (ablation) | EXP + A-CE + SM on CIFAR-FS · CIFAR-FS | Robust accuracy: 40.12 -> 40.29 | No clear effect |
Surprising findings
- Plain cross-entropy with CLIP anchors works better than cos-theta or Euclidean anchor objectives that are standard for anchor-based AT, because the softmax relaxes the alignment requirement when anchors are highly similar.
- TeCoA's failure under strong perturbations is attributed to the high CoS problem of CLIP anchors, a problem previously unnoticed in CLIP-based robustness work.
Limitations
- Performance is weak on Cifar100, DTD and SUN, attributed to low resolution and large inter-class differences.
- Zero-shot results rely on text names of test categories, so comparisons with few-shot methods are not like-for-like.
- Large-scale image-text adversarial pretraining like CLIP was not attempted due to data and compute limits.
Experimental setup
Datasets: CIFAR-FS MiniImageNet ImageNet-1K Cifar100
Benchmarks: CIFAR-FS MiniImageNet AwA2 aPY COCO Object STL10 OxfordPet DTD Caltech101 Caltech256 SUN Cifar100
Metrics: clean accuracy robust accuracy under FGSM, PGD-20, CW, AutoAttack
Baselines: AQ R-MAML GR TeCoA TRADES ITS-MAML
Builds on: CLIP TeCoA MMC loss Hypersphere Embedding (cos theta) TRADES GR
Closest papers by meaning
- Few-Shot Adversarial Prompt Learning on Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.88)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.88)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Boosting Adversarial Robustness with CLAT: Criticality Leveraged Adversarial Training (ICML 2025 · similarity 0.86)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.