Research map / Architectures and efficiency / Attention and transformers
Understanding Robustness of Transformers for Image Classification
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, Andreas Veit
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 only | ViT-L/16 · ImageNet-C | accuracy: 49.31% -> 76.08% | Improved |
| pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 only | ViT-L/16 · ImageNet-A | accuracy: 6.68% -> 44.00% | Improved |
| pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 only | ViT-L/16 · ImageNet-R | accuracy: 17.87% -> 65.28% | Improved |
| use ViT-L/16 instead of ResNet-101x3 when trained only on ILSVRC-2012 | ViT-L/16 · ImageNet-C | accuracy: 53.36% -> 49.31% | Hurt |
| use ViT-H/14 instead of ResNet-152x4 with JFT-300M pre-training | ViT-H/14 · ImageNet-A | accuracy: 51.23% -> 55.85% | Improved |
| use ViT-H/14 instead of ResNet-152x4 with JFT-300M pre-training | ViT-H/14 · ImageNet-C | accuracy: 73.41% -> 78.74% | Improved |
| use larger patch size 32x32 instead of 16x16 in ViT | ViT-L · Conflict Stimuli shape accuracy (JFT-300M) | accuracy: 53.65% -> 61.91% | Improved |
| scale ViT model size under JFT-300M pre-training | ViT · ImageNet-C | accuracy: 67.70% -> 78.74% | Improved |
| PGD perturbations computed on ViT applied to ResNet (transfer) | ViT-B/16 vs RN-101x1 · ILSVRC-2012 val subset | accuracy: 79.7% (-2.5) | No clear effect |
| remove all inter-patch attention in final blocks only (ablation) | ViT-B/16 · ILSVRC-2012 val | accuracy | No clear effect |
| remove inter-patch attention in initial blocks (ablation) | ViT-B/16 · ILSVRC-2012 val | accuracy | Hurt |
| remove self-attention layers vs MLP layers (lesion) (ablation) | ViT · ILSVRC-2012 val | accuracy | Mixed |
Closest papers by meaning
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.95)
- An Impartial Take to the CNN vs Transformer Robustness Contest (ECCV 2022 · similarity 0.93)
- Are Transformers more robust than CNNs? (NeurIPS 2021 · similarity 0.93)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.90)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.88)
- Revealing the Dark Secrets of Extremely Large Kernel ConvNets on Robustness (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
