Research map / Adversarial robustness and security / Certified and empirical robustness
Towards Understanding and Improving Adversarial Robustness of Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that softmax in ViT attention blocks causes floating-point underflow and gradient masking, and proposes the AAS attack (LPIPS-driven search over pre-softmax scaling factors) and AAS-AT adversarial training that plugs into existing AT methods.
Problem
Existing white-box attacks (PGD, CW, GAMA) are weaker on vision transformers than on CNNs, and adversarial training of ViTs is hard to stabilize; the paper asks why.
Approach
Attributes ViT gradient masking to floating-point underflow from large pre-softmax attention scores, and proposes automatically finding per-block pre-softmax scaling factors by maximizing an LPIPS-style feature distance between normal and rescaled models, then using these scaled models for attacks and periodically during adversarial training.
Key results
- On ViT-B16, AAS combined with GAMA improves attack strength by 2.2% on CIFAR-10 and 2.89% on CIFAR-100 (ImageNet-100 ~2.2% over GAMA).
- AAS-AT combined with PGD-AT, TRADES, and related methods improves AutoAttack robust accuracy by up to 3.47% on CIFAR-10 and 2.38% on CIFAR-100 (Trades+AWP).
- On ImageNet-100, AAS-AT with TRADES improves Auto-Attack robust accuracy from 26.46% to 28.04%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Adaptive Attention Scaling to GAMA | GAMA on ViT-B16 · CIFAR10 | robust accuracy: 59.78 -> 57.61 | Helped |
| Add Adaptive Attention Scaling to GAMA | GAMA on ViT-B16 · CIFAR100 | robust accuracy: 28.97 -> 26.08 | Helped |
| Add Adaptive Attention Scaling to GAMA | GAMA on ViT-B16 · ImageNet-100 | robust accuracy: 28.07 -> 25.64 | Helped |
| Add Adaptive Attention Scaling to PGD-100 | PGD-100 on ViT-B16 · CIFAR10 | robust accuracy: 61.10 -> 58.01 | Helped |
| Add Adaptive Attention Scaling to PGD-100 | PGD-100 on ViT-B16 · CIFAR100 | robust accuracy: 30.01 -> 27.02 | Helped |
| Add AAS-AT to Trades | Trades on ViT-B16 · ImageNet-100 | Auto-Attack accuracy: 26.46 -> 28.04 | Helped |
| Add AAS-AT to PGD-AT | PGD-AT on ViT-B16 · CIFAR10 | AutoAttack accuracy: 53.14 -> 56.61 | Helped |
| Add AAS-AT to Trades + AWP | Trades + AWP on ViT-B16 · CIFAR100 | AutoAttack accuracy: 27.03 -> 29.41 | Helped |
| Use LPIPS rather than cross-entropy to perturb scaling factors (ablation) | AAS attack · CIFAR10 | PGD-100 robust accuracy: 60.13 -> 58.01 | Helped |
| Use LPIPS rather than cross-entropy to perturb scaling factors (ablation) | AAS attack · CIFAR100 | PGD-100 robust accuracy: 29.11 -> 27.02 | Helped |
| Scale more attention blocks with AAS (ablation) | AAS attack on ViT-B16 · CIFAR10 | robust accuracy | Helped |
| Increase the number of attention blocks in the model (ablation) | AAS attack · CIFAR10 | robust accuracy | Mixed |
Surprising findings
- Simply scaling pre-softmax outputs manually by a single factor can already increase PGD-100 attack strength on ViTs, supporting the underflow hypothesis.
- Standard losses (cross-entropy, max-margin, GAMA) are less effective than LPIPS for finding pre-softmax scaling factors.
Limitations
- Gains over strongest attacks (AutoAttack) remain below AAS, which is still weaker than AutoAttack by under 1%.
- Improvements are small (typically 1-3%) and evaluated on CIFAR and a 100-class ImageNet subset only.
- Scaling factors are shared across heads/blocks in the manual baseline; per-block tuning was not performed due to cost.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1K (initialization) DDPM synthetic data
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100
Metrics: clean accuracy robust accuracy
Baselines: FGSM PGD-100 CW DLR GAMA AutoAttack PGD-AT TRADES AWP ART MART XCiT-S12 adversarial training
Builds on: PGD GAMA TRADES AWP Yu and Xu scaling analysis
Closest papers by meaning
- Boosting the Transferability of Adversarial Attack on Vision Transformer with Adaptive Token Tuning (NeurIPS 2024 · similarity 0.91)
- Enhancing Transferable Adversarial Attacks on Vision Transformers through Gradient Normalization Scaling and High-Frequency Adaptation (ICLR 2024 · similarity 0.91)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.90)
- Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability (NeurIPS 2025 · similarity 0.89)
- Attacking Transformers with Feature Diversity Adversarial Perturbation (AAAI 2024 · similarity 0.88)
- Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.