Research map / Architectures and efficiency / Attention and transformers
Vision Transformers Are Robust Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViT L-16 instead of BiT-m r101x3 | Image classification · ImageNet-A | top-1 accuracy: 28.10% and 4.3x higher | Helped |
| Use ViT L-16 instead of BiT-m r101x3 | Image classification · ImageNet-9 Mixed-Rand | top-1 accuracy: 76.62 -> 81.68 | Helped |
| Pre-train ViT B/16 on ImageNet-21k rather than ImageNet-1k (ablation) | ViT B/16 · ImageNet-A | top-1 accuracy: 8.630994 -> 21.746947 | Helped |
| Pre-train ViT B/16 on ImageNet-21k rather than ImageNet-1k (ablation) | ViT B/16 · ImageNet-R | top-1 accuracy: 28.213835 -> 41.815233 | Helped |
| Pre-train ViT B/16 on ImageNet-21k rather than ImageNet-1k (ablation) | ViT B/16 · ImageNet-O | AUPR: 26.25 -> 54.61 | Helped |
| Increase random Cutout masking from 0 to 0.5 (ablation) | ViT L-16 · ImageNet-1k validation, 1000 images | top-1 accuracy: 83 -> 60.4 | Hurt |
| Increase random Cutout masking from 0 to 0.5 (ablation) | BiT m-r101x3 · ImageNet-1k validation, 1000 images | top-1 accuracy: 79 -> 52 | Hurt |
| Use ViT L-16 rather than BiT-m r101x3 under random Cutout masking | Image classification · ImageNet-1k validation, masking factors 0, 0.05, 0.1, 0.2, 0.5 | top-1 accuracy: 79 vs 83; 76 vs 82.3; 75 vs 81.4; 72.4 vs 77.9; 52 vs 60.4 | Helped |
| Use ViT L-16 instead of ResNet-50 | Image classification · ImageNet-9 vulnerable-foreground challenge | challenge accuracy: 22.3 vs 20.02 | No clear effect |
| Use ViT rather than BiT for common perturbations | Image classification · ImageNet-P | mean flip rate | Helped |
| Use ViT rather than BiT for common corruptions | Image classification · ImageNet-C severity 5 | top-1 accuracy | Mixed |
| Use ViT rather than BiT for distribution shifts | Image classification · ImageNet-R | top-1 accuracy | Helped |
Closest papers by meaning
- Towards Robust Vision Transformer (CVPR 2022 · similarity 0.93)
- Discrete Representations Strengthen Vision Transformer Robustness (ICLR 2022 · similarity 0.93)
- Are Transformers more robust than CNNs? (NeurIPS 2021 · similarity 0.93)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.91)
- Improving Interpretation Faithfulness for Vision Transformers (ICML 2024 · similarity 0.89)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.