Research map / Architectures and efficiency / Attention and transformers
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAM to ViT-B/16 | ViT-B/16 · ImageNet | top-1 accuracy (%): 74.6 -> 79.9 | Helped |
| Add SAM to Mixer-B/16 | Mixer-B/16 · ImageNet | top-1 accuracy (%): 66.4 -> 77.4 | Helped |
| Add SAM to ViT-B/16 | ViT-B/16 · ImageNet-C | accuracy (%): +9.9% | Helped |
| Add SAM to ViT-B/16 | ViT-B/16 · ImageNet-C | accuracy (%): 46.6 -> 56.5 | Helped |
| Remove softmax from self-attention (ablation) | ViT-B/16 · ImageNet | top-1 accuracy (%): 74.6% vs. 70.5% | Hurt |
| Use strong augmentations instead of SAM (ablation) | ViT-B/16 · ImageNet (1/10 training set) | top-1 accuracy (%): 7.6% | Hurt |
| Combine SAM with strong augmentations (ablation) | ViT-B/16 · ImageNet | top-1 accuracy (%): 79.9 -> 81.5 | Helped |
| Train ViT-B/16 with SGD + SAM instead of SGD alone (ablation) | ViT-B/16 · ImageNet | accuracy (%): 71.5% -> 79.1% | Helped |
| Increase SAM perturbation strength from rho 0.05 to rho 0.2 (ablation) | ViT-B/16 · ImageNet | top-1 accuracy (%): 77.5 -> 79.9 | Helped |
| Increase SAM perturbation strength from rho 0.2 to rho 0.25 (ablation) | ViT-B/16 · ImageNet | top-1 accuracy (%): 79.9 -> 79.3 | Hurt |
| Train with twice the vanilla schedule (600 epochs) (ablation) | ViT and MLP-Mixer · ImageNet | clean accuracy | No clear effect |
| Add SAM during supervised contrastive learning (ablation) | ViT-B/16 · ImageNet | top-1 accuracy (%): 77.4% to 80.0% | Helped |
Closest papers by meaning
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.91)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.90)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.89)
- Improving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection (AAAI 2024 · similarity 0.88)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.88)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.