Research map / Adversarial robustness and security / Certified and empirical robustness
Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Exploits computational redundancy in ViTs (attention sparsity, head permutation, clean tokens, ghost MoE, test-time robust tokens, and a policy-gradient online scheduler) to boost transferable adversarial attacks on ImageNet.
Problem
Adversarial examples crafted on ViTs transfer well, but it is unclear how ViT computational redundancy can be harnessed to further improve black-box transferability of attacks across CNNs, ViTs and VLLMs.
Approach
Treats data-level and model-level redundancy in ViTs as a resource: randomly sparsifies attention, permutes QK heads, injects clean tokens, uses dropout-based ghost MoE FFNs, and learns robustification tokens at test time, with a REINFORCE-learned per-block policy choosing operations online.
Key results
- Ours with ViT-B/16 surrogate reaches 86.9% average fooling rate vs 77.8% for PGN, and 99.7% on ViT-B/16 target.
- With PiT-B surrogate, average fooling rate 87.4% vs 75.1% for PGN; with Swin-T surrogate 88.9% vs 85.3% for PGN.
- Attacks on VLLMs (LLaVA, Qwen2.5-VL, InternVL2.5, DeepSeek-VL) average 2.2% improvement over runner-up PGN, with 5.5% gain on Qwen and 2.6% on InternVL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned redundancy operations and robust tokens instead of PGN | ViT-B/16 surrogate transfer attack · ImageNet-1k transfer, average across target models | fooling rate: 86.9% vs. 77.8% | Helped |
| Use learned redundancy operations and robust tokens instead of PGN | PiT-B surrogate transfer attack · ImageNet-1k transfer, average across target models | fooling rate: 87.4% vs. 75.1% | Helped |
| Use learned redundancy operations and robust tokens instead of PGN | Swin-T surrogate transfer attack · ImageNet-1k transfer, average across target models | fooling rate: 88.9% vs. 85.3% | Helped |
| Apply moderate attention-weight sparsification (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | black-box attack success rate | Mixed |
| Increase attention sparsification beyond the moderate range (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | attack success rate | Hurt |
| Permute attention heads in selected layers (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | adversarial transferability | Helped |
| Use excessive head permutation (p = 0.5, r = 1.0) (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | adversarial transferability | Hurt |
| Add a small proportion of clean tokens (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | attack success rate | Helped |
| Add excessive clean tokens (r > 0.4) (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | attack success rate | Mixed |
| Increase the number of ghost MoE experts (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | adversarial transferability | Helped |
| Use neuron drop rate 0.3 in ghost MoE (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | adversarial transferability | Helped |
| Use excessive ghost MoE dropout (0.5) (ablation) | MI-FGSM with ViT surrogate · ImageNet-1k transfer | adversarial transferability | Hurt |
Surprising findings
- Global robust tokens are ineffective with few tokens (28.45% with 1 token, below the 54.09% MI-FGSM baseline in the ablation), only helping at 100+ tokens.
Limitations
- Trade-off between number of robust tokens, computational overhead, and transferability; excessive attention sparsity or clean tokens degrade performance.
- Swin and PiT require architecture-specific workarounds for robust token insertion.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: attack success rate fooling rate average fooling rate
Baselines: MI-FGSM NI-FGSM EMI-FGSM VMI-FGSM PGN DTA TGR GNS FPR
Builds on: MI-FGSM PGN VMI-FGSM NI-FGSM EMI-FGSM
Closest papers by meaning
- Boosting the Transferability of Adversarial Attack on Vision Transformer with Adaptive Token Tuning (NeurIPS 2024 · similarity 0.93)
- Improving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection (AAAI 2024 · similarity 0.92)
- Attacking Transformers with Feature Diversity Adversarial Perturbation (AAAI 2024 · similarity 0.90)
- Learning to Transform Dynamically for Better Adversarial Transferability (CVPR 2024 · similarity 0.90)
- Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement (CVPR 2025 · similarity 0.90)
- Enhancing Adversarial Transferability with Adversarial Weight Tuning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.