Research map / Adversarial robustness and security / Certified and empirical robustness
Are Transformers more robust than CNNs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use augmentation warm-up during adversarial training of DeiT-S | DeiT-S · ImageNet val, PGD-5 | robust accuracy: ~4% -> ~44% | Helped |
| Replace ReLU with GELU (ablation) | ResNet-50 · ImageNet val, PGD-100 | robust accuracy: 32.26 -> 40.27 (+8.01%) | Helped |
| Replace ReLU with GELU (ablation) | ResNet-50 · ImageNet val, AutoAttack | robust accuracy: 26.41 -> 35.51 | Helped |
| Add CutMix (ablation) | ResNet-50 · ImageNet val, Texture Patch Attack | robust accuracy: +29.4% | Helped |
| Add RandAug and CutMix (ablation) | ResNet-50 · ImageNet val, Texture Patch Attack | robust accuracy: +32.7% | Helped |
| Use cosine instead of step-decay learning-rate scheduling (ablation) | ResNet-50 · ImageNet-A | accuracy: +0.1% | Helped |
| Replace SGD with AdamW (ablation) | ResNet-50 · ImageNet-A | accuracy: -0.2% | Hurt |
| Add RandAug and Mixup (ablation) | ResNet-50 · ImageNet-A | accuracy: +3.0% | Helped |
| Add RandAug and Mixup (ablation) | ResNet-50 · ImageNet-C | mCE: +4.6 | Helped |
| Adopt the full DeiT training recipe (ablation) | ResNet-50 · ImageNet-A | accuracy: 3.2 -> 4.5 (+1.3%) | Helped |
| Distill DeiT-S predictions into ResNet-50 (ablation) | ResNet-50 · ImageNet-A | accuracy: 12.2 -> 5.2 (-7.0) | Hurt |
| Distill ResNet-50* predictions into DeiT-S (ablation) | DeiT-S · ImageNet-A | accuracy: 4.5 -> 10.9 (+6.4) | Helped |
Closest papers by meaning
- Can CNNs Be More Robust Than Transformers? (ICLR 2023 · similarity 0.95)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.90)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.90)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.88)
- Curved Representation Space of Vision Transformers (AAAI 2024 · similarity 0.88)
- ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.