Research map / Adversarial robustness and security / Black-box and transfer attacks
Boosting the Transferability of Adversarial Attack on Vision Transformer with Adaptive Token Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Token Tuning (ATT), a transfer-based attack on ViTs combining adaptive token-gradient re-scaling, self-paced semantic-guided patch out, and hybrid attention truncation to improve black-box transferability.
Problem
Transfer-based adversarial attacks on vision transformers have limited transferability because existing token-gradient regularization zeroes out the largest token gradients, discarding feature information and overfitting the surrogate model's attention interactions.
Approach
Instead of zeroing the largest token gradients, mildly re-scale them by a penalty factor, adapt that re-scaling using gradient variance ratios across consecutive layers, replace random PatchOut with feature-importance-guided self-paced patch discarding, and truncate attention gradients in deep layers (hybrid soft/hard truncation).
Key results
- ATT achieves 77.4% average black-box ASR on ViTs without input diversity, versus 69.8% for TGR (Table 1).
- With self-paced patch out, ATT reaches 54.1% average ASR on CNNs and 38.7% on defended CNNs for ViT-B surrogate; for CaiT-S/24 surrogate it reaches 58.3% on defended CNNs.
- The paper reports an average 10.1% improvement in attack success over state-of-the-art transfer-based attacks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random PatchOut with self-paced patch out under semantic guidance (ablation) | ATT+PO · ViTs | ASR (%): 77.1 -> 80.3 | Helped |
| Replace random PatchOut with self-paced patch out under semantic guidance (ablation) | ATT+PO · CNNs | ASR (%): 51.7 -> 54.1 | Helped |
| Replace random PatchOut with self-paced patch out under semantic guidance (ablation) | ATT+PO · Def-CNNs | ASR (%): 37.1 -> 38.7 | Helped |
| Add adaptive variance reduction, soft truncation, and hard truncation (ablation) | ATT without these strategies · ViTs (ViT-B/16 surrogate) | ASR (%): 49.9 -> 80.3 | Helped |
| Add adaptive variance reduction, soft truncation, and hard truncation (ablation) | ATT without these strategies · CNNs (ViT-B/16 surrogate) | ASR (%): 29.1 -> 54.1 | Helped |
| Add adaptive variance reduction, soft truncation, and hard truncation (ablation) | ATT without these strategies · Def-CNNs (ViT-B/16 surrogate) | ASR (%): 19.3 -> 38.7 | Helped |
| Use adaptive rather than fixed variance reduction (ablation) | ATT variance-reduction strategy · ViTs (ViT-B/16 surrogate) | ASR (%): 72.5 -> 73.5 | Helped |
| Use adaptive rather than fixed variance reduction (ablation) | ATT variance-reduction strategy · CNNs (ViT-B/16 surrogate) | ASR (%): 46.2 -> 47.5 | Helped |
| Use adaptive rather than fixed variance reduction (ablation) | ATT variance-reduction strategy · Def-CNNs (ViT-B/16 surrogate) | ASR (%): 33.0 -> 33.6 | Helped |
Surprising findings
- Visformer-S, a shallow model, achieved its best result without attention truncation, suggesting gradient scaling alone suffices for it.
- Attacking the MLP module of CaiT-S/24 gave better transferability than attacking attention in that case, contrary to the general trend.
Limitations
- Relationship between gradient variance reduction and transferability lacks theoretical support, as acknowledged by the authors.
- Hyperparameters (gamma, lambda, alpha, beta, truncation factors) are tuned per surrogate model, and the truncation strategy depends on model depth.
Experimental setup
Datasets: ILSVRC2012 validation set
Benchmarks: ImageNet (ILSVRC2012 subset) ViT black-box models undefended CNNs defended CNNs robust ViTs object detection (qualitative) semantic segmentation (qualitative)
Metrics: attack success rate average attack success rate average iterations to misclassification
Baselines: MIM VMI SGM PNA TGR PatchOut
Builds on: TGR PatchOut MI-FGSM VMI-FGSM SGM
Closest papers by meaning
- Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability (NeurIPS 2025 · similarity 0.93)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.91)
- Improving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection (AAAI 2024 · similarity 0.91)
- Attacking Transformers with Feature Diversity Adversarial Perturbation (AAAI 2024 · similarity 0.91)
- Enhancing Adversarial Transferability with Adversarial Weight Tuning (AAAI 2025 · similarity 0.91)
- Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.