Research map / Adversarial robustness and security / Certified and empirical robustness
Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Vision Transformers (ViT-B/2, ViT-L/2) trained with TRADES adversarial training on EDM-generated synthetic data outperform WideResNet (WRN-70-16) on CIFAR-10 and CIFAR-100 AutoAttack robust accuracy.
Problem
Whether Vision Transformers can match or surpass WideResNets in adversarial robustness on small-scale datasets like CIFAR-10 and CIFAR-100, where WRN has dominated RobustBench leaderboards.
Approach
Combine ViT architectures with diffusion-generated synthetic data (EDM, 50M samples from Wang et al.) in TRADES adversarial training, using Lion optimizer, label smoothing 0.4, and no early stopping, to show ViTs beat WRNs in robust accuracy.
Key results
- ViT-L/2 reaches 74.97% AutoAttack robust accuracy on CIFAR-10 (l_inf, eps=8/255), +3.9% over prior SOTA per abstract, with clean accuracy 95.76%.
- ViT-L/2 reaches 44.07% AA robust accuracy on CIFAR-100 (l_inf, eps=8/255), +1.4% over prior SOTA.
- ViT-B/2 (85M params) reaches 71.31% AA on CIFAR-10 after 2000 epochs, surpassing WRN-70-16 (70.69%) with about 3x fewer parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ViT-L/2 with the proposed synthetic-data adversarial-training approach rather than the strongest WRN baseline | CIFAR-10 (ℓ∞, ϵ = 8/255) · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.07 -> 74.94 | Helped |
| Train ViT-L/2 with the proposed synthetic-data adversarial-training approach rather than the strongest WRN baseline | CIFAR-10 (ℓ2, ϵ = 0.5) · CIFAR-10 (ℓ2, ϵ = 0.5) | AA accuracy: 84.86 -> 86.06 | Helped |
| Train ViT-L/2 with the proposed synthetic-data adversarial-training approach rather than the strongest WRN baseline | CIFAR-100 (ℓ∞, ϵ = 8/255) · CIFAR-100 (ℓ∞, ϵ = 8/255) | AA accuracy: 42.67 -> 44.07 | Helped |
| Increase label smoothing from 0.0 to 0.1 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 70.52 -> 71.07 | Helped |
| Vary label smoothing across 0.1, 0.2, 0.3, and 0.4 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.07; 70.88; 71.01; 70.98 | No clear effect |
| Set TRADES β to 3 rather than 5 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 70.98 -> 71.31 | Helped |
| Vary TRADES β among 2, 4, and 6 rather than using β = 3 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.31 -> 71.22; 71.06; 70.69 | Hurt |
| Increase the generated-data mix ratio from 0.0 to 0.9 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255), PGD-40 | AA accuracy: 0.5685 -> 0.7377 | Helped |
| Increase the generated-data mix ratio from 0.9 to 1.0 (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255), PGD-40 | AA accuracy: 0.7377 -> 0.7356 | Hurt |
| Increase ViT-B/2 training from 2000 to 6000 epochs (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.31 -> 72.52 | Helped |
| Scale the model from ViT-B/2 to ViT-L/2 at 2000 epochs (ablation) | ViT · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.31 -> 72.17 | Helped |
| Replace Lion with SGD (ablation) | ViT-B/2 · CIFAR-10 (ℓ∞, ϵ = 8/255) | AA accuracy: 71.31 -> ✗ | Hurt |
Surprising findings
- ViT performed best with TRADES beta=3 on CIFAR-10, not beta=5 as optimal for WRN, and a 0.9 synthetic mix ratio rather than 0.8.
- Standard SGD training of ViT-B/2 yielded only 75.06% clean accuracy and failed to achieve robustness, while Lion gave 94.35% clean and 71.31% AA.
Limitations
- Did not explore other transformer architectures such as Swin Transformer.
- Training is computationally expensive (TPU v4-256), limiting accessibility and reproducibility.
- Comparisons use Wang et al.'s 50M EDM data; Bartoldson et al. data was unavailable, so the comparison is not on equal footing.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 EDM-generated synthetic data (50M, Wang et al. 2023)
Benchmarks: CIFAR-10 CIFAR-100 RobustBench
Metrics: clean accuracy AutoAttack robust accuracy PGD-40 robust accuracy FLOPs
Baselines: WRN-70-16 (Wang et al. 2023) RaWRN-70-16 (Peng et al. 2023) WRN-28-10 (Gowal et al. 2021) WRN-28-10 (Pang et al. 2022) WRN-70-16 (Rebuffi et al. 2021) WRN-94-16 (Bartoldson et al. 2024)
Builds on: TRADES EDM Wang et al. 2023 diffusion adversarial training Lion optimizer AutoAttack
Closest papers by meaning
- When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture (NeurIPS 2022 · similarity 0.92)
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.91)
- Benign Overfitting in Adversarial Training for Vision Transformers (ICML 2026 · similarity 0.91)
- Towards Robust Vision Transformers: Path Dependency Analysis and a Simple Two-Stage Adversarial Training (CVPR 2026 · similarity 0.90)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.90)
- Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.