Research map / Detection and segmentation / CNN features
Understanding The Robustness in Vision Transformers
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, Jose M. Alvarez
This paper tried a common change: More attention heads
Increase the number of attention heads · ImageNet robustness · robustness Mixed
What 107 accepted papers report for this change
- 62 improved
- 13 no clear effect
- 13 mixed
- 19 hurt
- Universal Approximation with Softmax Attention ICML 2026 · increase the number of attention heads H · synthetic truncated linear model (d=10, n=50) · test MSEImproved
- TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning ICML 2026 · raise number of heads from 10 to 14 · ScienceQA SIG detection · F1: 89.74 -> 77.09Hurt
- Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing ICML 2026 · increase number of attention heads from 1 to 8 · UEA classification (3 datasets) · error rate: error rates reduced as M increases from 1 to 8; plateau around M = 4 for GPImproved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FAN-ECA channel processing instead of SE channel attention (ablation) | FAN-ViT-S · ImageNet-C | IN-C accuracy: 62.0 -> 64.6 | Improved |
| Use FAN-ECA channel processing instead of self-attention channel processing (ablation) | FAN-ViT-S · ImageNet-C | IN-C accuracy: 61.7 -> 64.6 | Improved |
| Replace conventional transformer blocks with FAN blocks (ablation) | ViT-S* · ImageNet-C | IN-C accuracy: 58.1 -> 61.7 | Improved |
| Add FAN blocks to shifted-window Swin-T (ablation) | Swin-T · ImageNet-C | IN-C accuracy: 55.4 -> 59.4 | Improved |
| Add FAN to ConvNeXt-T (ablation) | ConvNeXt-T · ImageNet-C | IN-C accuracy: 59.1 -> 60.8 | Improved |
| Add the DeiT training recipe (ablation) | ViT-S · ImageNet-C | IN-C accuracy: 54.2 -> 57.1 | Improved |
| Increase ViT-S depth from 8 to 12 blocks (ablation) | ViT-S with DeiT recipe · ImageNet-C | IN-C accuracy: 57.1 -> 58.0 | Improved |
| Add knowledge distillation (ablation) | ViT-S with 12 blocks · ImageNet-C | IN-C accuracy: 58.0 -> 59.6 | Improved |
| Use ImageNet-22K pretraining without KD (ablation) | ViT-S with 12 blocks · ImageNet-C | IN-C accuracy: 58.0 -> 59.7 | Improved |
| Add the DeiT training recipe (ablation) | ResNet-50 · ImageNet-C | IN-C accuracy: 38.8 -> 43.9 | Improved |
| Add SE channel attention (ablation) | ResNet-50 with DeiT recipe · ImageNet-C | IN-C accuracy: 43.9 -> 50.1 | Improved |
| Use strided convolution for downsampling (ablation) | ResNet-50 with DeiT recipe and SE · ImageNet-C | IN-C accuracy: 50.1 -> 52.1 | Improved |
Closest papers by meaning
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.90)
- An Impartial Take to the CNN vs Transformer Robustness Contest (ECCV 2022 · similarity 0.90)
- Robustifying Token Attention for Vision Transformers (ICCV 2023 · similarity 0.90)
- Towards Robust Vision Transformer (CVPR 2022 · similarity 0.89)
- Can CNNs Be More Robust Than Transformers? (ICLR 2023 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
