Research map / Adversarial robustness and security / Black-box and transfer attacks
Attacking Transformers with Feature Diversity Adversarial Perturbation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FDAP, a label-free white-box attack on ViTs that maximizes the high-frequency (residual after direct component) diversity of middle-layer norm2 features, achieving higher transferability to ViTs, CNNs, MLPs and other tasks.
Problem
Adversarial attacks on Vision Transformers rely on classification labels and transfer poorly to other architectures and tasks.
Approach
Replace label-based cross-entropy with a label-free feature diversity loss that measures the norm of the residual after removing the direct (rank-1) component of intermediate features, applied to CKA-selected middle-layer norm2 outputs, to accelerate feature collapse and boost transferability.
Key results
- On black-box ViTs with ViT-L/16 surrogate, FDAP (Ours) reaches 54.7% fooling on DeiT-Dis and 69.0% on CaiT-XXS, and MIM+Ours is higher still (e.g., 76.2% on ResMLP-24).
- With DeiT-B surrogate, transfer to Mixer-B rises markedly versus baselines; the paper reports roughly doubling of fooling rate on Mixer-B compared with other methods when ConViT-B is the surrogate.
- Cross-task attacks from DeiT surrogates reduce DETR mAP from 44.03 to about 15 and SegFormer mIoU from 78.56 to about 30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FDAP instead of PNA | ViT-L/16 attack · ImageNet 2012, DeiT-Dis | fooling rate: 41.9 -> 54.7 | Helped |
| Use FDAP instead of PNA | ViT-L/16 attack · ImageNet 2012, Res50 | fooling rate: 36.7 -> 37.3 | Helped |
| Use FDAP instead of PNA | ViT-L/16 attack · ImageNet 2012, Mixer-B | fooling rate: 52.4 -> 64.6 | Helped |
| Combine MIM with FDAP | ViT-L/16 attack · ImageNet 2012, DeiT-Dis | fooling rate: 54.7 -> 59.1 | Helped |
| Combine MIM with FDAP | ViT-L/16 attack · ImageNet 2012, Res50 | fooling rate: 37.3 -> 40.0 | Helped |
| Combine MIM with FDAP | ViT-L/16 attack · ImageNet 2012, Mixer-B | fooling rate: 64.6 -> 68.3 | Helped |
| Apply FDAP perturbations generated on DeiT-T | cross-task transfer to DETR · COCO 2017, object detection | mAP: 44.03 -> 15.20 | Hurt |
| Apply FDAP perturbations generated on DeiT-T | cross-task transfer to SegFormer · Cityscapes, semantic segmentation | mIoU: 78.56 -> 31.17 | Hurt |
| Apply FDAP perturbations generated on DeiT-T | cross-task transfer to ViTPose-B · COCO 2017, pose estimation | AP: 75.53 -> 44.32 | Hurt |
| Apply FDAP perturbations generated on DeiT-T | cross-task transfer to Swin-v2-MIM · NYUV2 val, depth estimation | Abs Rel: 0.089 -> 0.122 | Hurt |
Surprising findings
- The method is label-free yet transfers to CNNs, MLPs, and models for other tasks such as segmentation and depth estimation.
Limitations
- Relies on access to model parameters and structure for generating attacks (white-box).
- Less effective than label-based methods when surrogate and victim are the same model.
- Hyperparameters (e.g., accelerating parameter beta, layer range) require per-architecture tuning.
Experimental setup
Datasets: ImageNet 2012 validation subset (1000 images) COCO 2017 Cityscapes NYUv2
Benchmarks: ImageNet 2012 validation (fooling rate) COCO 2017 Cityscapes NYUv2
Metrics: fooling rate mAP degradation mIoU/mAcc degradation AP/AR degradation Abs Rel Sq Rel
Baselines: MIM DIM NIM Patch-Fool SE SAGA ATA PNA
Builds on: MI-FGSM DI-FGSM NI-FGSM Centered Kernel Alignment
Closest papers by meaning
- Boosting the Transferability of Adversarial Attack on Vision Transformer with Adaptive Token Tuning (NeurIPS 2024 · similarity 0.91)
- Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability (NeurIPS 2025 · similarity 0.90)
- Improving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection (AAAI 2024 · similarity 0.89)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.88)
- Random Entangled Tokens for Adversarially Robust Vision Transformer (CVPR 2024 · similarity 0.88)
- Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.