Research map / Adversarial robustness and security / Attacks on text and graphs
Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TGA-ZSR, which adversarially fine-tunes CLIP ViT-B/32 on Tiny-ImageNet with text-guided attention alignment losses to improve zero-shot adversarial robustness while preserving clean accuracy.
Problem
Zero-shot adversarial robustness of vision-language models such as CLIP: adversarial fine-tuning on one dataset degrades clean accuracy and generalization to unseen datasets, and the mechanism by which attacks change model outputs is unclear.
Approach
Observes that adversarial perturbations shift text-guided attention (image features dotted with the ground-truth text embedding), and adds two losses: an Attention Refinement loss aligning adversarial attention of the target model to clean attention of the original CLIP, and an Attention-based Model Constraint loss keeping clean attention of the target close to the original model.
Key results
- Zero-shot robust accuracy averaged over 16 datasets under PGD-100 (eps 1/255): TGA-ZSR 42.09% vs PMG-AFT 32.51% (about 9.58 points higher).
- Zero-shot clean accuracy average 56.44% for TGA-ZSR vs 46.60% for PMG-AFT, but 59.85% for FARE, which is higher than TGA-ZSR.
- Under AutoAttack (eps 1/255) the average robust accuracy is 30.86% vs PMG-AFT 30.78%, a marginal gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TGA-ZSR attention refinement and model constraint to CLIP fine-tuning | CLIP ViT-B/32 · 16 datasets, PGD-100 | robust accuracy (average): 32.51 -> 42.09 | Helped |
| Use TGA-ZSR instead of FARE | CLIP ViT-B/32 · 16 datasets, clean | accuracy (average): 59.85 -> 56.44 | Hurt |
| Add attention refinement loss to cross-entropy adversarial fine-tuning (ablation) | CLIP ViT-B/32 · 16 datasets, PGD-100 | robust accuracy (average): 29.45 -> 31.71 | Helped |
| Add attention-based model constraint loss to cross-entropy plus attention refinement (ablation) | CLIP ViT-B/32 · 16 datasets, PGD-100 | robust accuracy (average): 31.71 -> 41.96 | Helped |
| Add attention refinement loss to cross-entropy adversarial fine-tuning (ablation) | CLIP ViT-B/32 · 16 datasets, clean | accuracy (average): 44.97 -> 49.96 | Helped |
| Add attention-based model constraint loss to cross-entropy plus attention refinement (ablation) | CLIP ViT-B/32 · 16 datasets, clean | accuracy (average): 49.96 -> 56.48 | Helped |
| Replace text-guided attention with vision-based Grad-CAM attention (ablation) | TGA-ZSR · 16 datasets, PGD-100 | robust accuracy (average): 41.96 -> 29.47 | Hurt |
| Use L2 distance instead of cosine distance in attention losses (ablation) | TGA-ZSR · 16 datasets, PGD-100 | robust accuracy (average): 29.57 -> 41.96 | Helped |
| Use L2 distance instead of L1 distance in attention losses (ablation) | TGA-ZSR · 16 datasets, PGD-100 | robust accuracy (average): 29.54 -> 41.96 | Helped |
| Set learning rate to 0.0001 instead of 0.001 (ablation) | TGA-ZSR · 16 datasets, PGD-100 | robust accuracy (average): 22.94 -> 41.96 | Helped |
| Set learning rate to 0.0001 instead of 0.00001 (ablation) | TGA-ZSR · 16 datasets, PGD-100 | robust accuracy (average): 24.36 -> 41.96 | Helped |
| Increase perturbation strengths across PGD evaluation to ε=1/255, 2/255 and 4/255 (ablation) | TGA-ZSR · 16 datasets, PGD at varied ε | robust accuracy (average) | Mixed |
Surprising findings
- Replacing text-guided attention with Grad-CAM vision-based attention yields results close to PMG-AFT, showing the gain depends on text guidance only partly.
- Under AutoAttack the improvement over PMG-AFT is negligible (30.86 vs 30.78 average), which the authors acknowledge as a limitation.
Limitations
- Gains are limited under the stronger AutoAttack, as the authors acknowledge.
- Clean accuracy is lower than FARE, which prioritizes clean accuracy.
- Attention computation adds about 15% training memory relative to PMG-AFT.
Experimental setup
Datasets: Tiny-ImageNet
Benchmarks: Tiny-ImageNet CIFAR-10 CIFAR-100 STL-10 SUN397 Food101 OxfordPets Flowers102 DTD EuroSAT FGVC-Aircraft ImageNet
Metrics: zero-shot robust accuracy zero-shot clean accuracy average accuracy
Baselines: CLIP FT-Clean FT-Adv. TeCoA FARE PMG-AFT
Builds on: TeCoA PMG-AFT FARE CLIP PGD adversarial training
Closest papers by meaning
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.91)
- Few-Shot Adversarial Prompt Learning on Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.89)
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness (CVPR 2024 · similarity 0.88)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
- Enhancing CLIP Robustness via Cross-Modality Alignment (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.