Research map / Adversarial robustness and security / Attacks on text and graphs
SlowFormer: Adversarial Attack on Compute and Energy Consumption of Efficient Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlowFormer, a universal adversarial patch that pastes onto images to increase the compute (FLOPs) of input-adaptive efficient vision transformers (A-ViT, ATS, AdaViT), and tests an adversarial-training defense.
Problem
Input-adaptive efficient vision transformers reduce computation per input, but an adversary can exploit this dependence to increase their compute and power consumption at inference time.
Approach
Optimize a single universal, image-agnostic adversarial patch (not per-image perturbations) that maximizes the efficient model's computation by attacking each method's compute-control objective (ponder loss for A-ViT, usage loss for AdaViT, attention-entropy loss for ATS), with model weights frozen.
Key results
- On A-ViT (ViT-Tiny), SlowFormer raises compute from 0.87 to 1.3 GFLOPs, achieving 100% Attack Success, using a 64x64 patch (8% of image area).
- On ATS with ViT-Small, SlowFormer reaches 60% Attack Success; on AdaViT with ViT-Small, 40% Attack Success.
- Adversarial training defense reduces A-ViT attacked compute from 1.26 to 1.01 GFLOPs (vs 0.87 unattacked), but does not fully remove the attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SlowFormer's universal patch | A-ViT · ImageNet-1K val, ViT-Tiny | Attack Success: 100% | Helped |
| apply SlowFormer's universal patch | A-ViT · ImageNet-1K val, ViT-Small | Attack Success: 99% | Helped |
| apply SlowFormer's universal patch | ATS · ImageNet-1K val, ViT-Tiny | Attack Success: 35% | Helped |
| apply SlowFormer's universal patch | ATS · ImageNet-1K val, ViT-Small | Attack Success: 60% | Helped |
| apply SlowFormer's universal patch | AdaViT · ImageNet-1K val, ViT-Small | Attack Success: 40% | Helped |
| apply SlowFormer's universal patch | A-ViT · CIFAR-10 val, ViT-Tiny | Attack Success: 41% | Helped |
| apply SlowFormer's universal patch | ATS · CIFAR-10 val, ViT-Tiny | Attack Success: 34.1% | Helped |
| include task loss to preserve classification accuracy during the compute attack (ablation) | A-ViT SlowFormer · ImageNet-1K val, ViT-Tiny | Top-1 accuracy: 4.7% -> 68.5% | Helped |
| vary patch size from 64 × 64 to 16 × 16 (ablation) | A-ViT SlowFormer · ImageNet-1K val, ViT-Tiny | Attack Success: 100% -> 27% | Hurt |
| randomize the patch location (ablation) | A-ViT SlowFormer · ImageNet-1K val | Attack Success: 100% | No clear effect |
| train A-ViT with a set of universal adversarial patches (ablation) | A-ViT · ImageNet-1K val, ViT-Tiny | Attack Success: 100% -> 34% | Hurt |
Surprising findings
- Standard NTAP and TAP accuracy-attack patches reduce compute of the efficient models instead of increasing it, especially for ATS.
- A 32x32 patch (2% of image area) still achieves 90% Attack Success on A-ViT-Tiny and 78% on A-ViT-Small.
Limitations
- Power consumption is not measured on hardware; FLOPs are used as a proxy.
- Attack Success on AdaViT and ATS is substantially lower than on A-ViT, and the attack is weaker on CIFAR-10 for ATS.
- The proposed adversarial-training defense only partially mitigates the attack.
Experimental setup
Datasets: ImageNet-1K CIFAR-10
Benchmarks: ImageNet-1K CIFAR-10
Metrics: Top-1 accuracy GFLOPs Attack Success
Baselines: Random Patch TAP NTAP Adversarial training defense
Builds on: A-ViT ATS AdaViT Patch-Fool universal adversarial patch
Closest papers by meaning
- Understanding and Defending Patched-based Adversarial Attacks for Vision Transformer (ICML 2023 · similarity 0.88)
- Boosting the Transferability of Adversarial Attack on Vision Transformer with Adaptive Token Tuning (NeurIPS 2024 · similarity 0.87)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.86)
- DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms (NeurIPS 2024 · similarity 0.85)
- Attacking Transformers with Feature Diversity Adversarial Perturbation (AAAI 2024 · similarity 0.85)
- A Unified, Resilient, and Explainable Adversarial Patch Detector (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.