AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Understanding Robustness of Transformers for Image Classification

Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, Andreas Veit

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 462 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 onlyViT-L/16 · ImageNet-Caccuracy: 49.31% -> 76.08%Improved
pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 onlyViT-L/16 · ImageNet-Aaccuracy: 6.68% -> 44.00%Improved
pre-train ViT-L/16 on JFT-300M instead of ILSVRC-2012 onlyViT-L/16 · ImageNet-Raccuracy: 17.87% -> 65.28%Improved
use ViT-L/16 instead of ResNet-101x3 when trained only on ILSVRC-2012ViT-L/16 · ImageNet-Caccuracy: 53.36% -> 49.31%Hurt
use ViT-H/14 instead of ResNet-152x4 with JFT-300M pre-trainingViT-H/14 · ImageNet-Aaccuracy: 51.23% -> 55.85%Improved
use ViT-H/14 instead of ResNet-152x4 with JFT-300M pre-trainingViT-H/14 · ImageNet-Caccuracy: 73.41% -> 78.74%Improved
use larger patch size 32x32 instead of 16x16 in ViTViT-L · Conflict Stimuli shape accuracy (JFT-300M)accuracy: 53.65% -> 61.91%Improved
scale ViT model size under JFT-300M pre-trainingViT · ImageNet-Caccuracy: 67.70% -> 78.74%Improved
PGD perturbations computed on ViT applied to ResNet (transfer)ViT-B/16 vs RN-101x1 · ILSVRC-2012 val subsetaccuracy: 79.7% (-2.5)No clear effect
remove all inter-patch attention in final blocks only (ablation)ViT-B/16 · ILSVRC-2012 valaccuracyNo clear effect
remove inter-patch attention in initial blocks (ablation)ViT-B/16 · ILSVRC-2012 valaccuracyHurt
remove self-attention layers vs MLP layers (lesion) (ablation)ViT · ILSVRC-2012 valaccuracyMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy