Research map / Adversarial robustness and security / Certified and empirical robustness
Revisiting Adversarial Training at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales adversarial training to ViT models up to ViT-g/14 and web-scale data (DataComp-1B) using a two-stage coarse-to-fine, weak-to-strong schedule with a frozen CLIP text encoder, setting new AutoAttack robustness records on ImageNet-1K.
Problem
Adversarial training is computationally expensive and has been confined to small models (e.g., ResNet-50) and small datasets (e.g., CIFAR-10), leaving its scalability to foundation-model scale unexplored.
Approach
A two-stage pipeline: lightweight pre-training with reduced image resolution (112px) and weak PGD-1 attacks, followed by short full-resolution fine-tuning with stronger PGD-3 attacks; plus a frozen CLIP text encoder providing classifier weights so web-crawled image-text data can be used for adversarial training.
Key results
- ViT-g/14 trained on DataComp-1B achieves 71.0% l-inf robust accuracy under AutoAttack on ImageNet-1K (4/255), surpassing prior best by about 11.4%.
- The same model reaches 70.4% l2 and 46.7% l1 robust accuracy, improving prior best l2 and l1 by about 14% and 13% despite training only for l-inf robustness.
- Two-stage schedule (112px PGD-1 pre-training then 224px PGD-3 fine-tuning) reduces compute with a reported up to 2x speedup and ~5x overall claimed speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the model from ViT-B/16 to ViT-L/16 | AdvXL trained on ImageNet-1K · ImageNet-1K | PGD-20 robust accuracy: 52.5 -> 54.7 | Helped |
| train ViT-L/16 with ImageNet-21K instead of ImageNet-1K | AdvXL ViT-L/16 · ImageNet-21K | PGD-20 robust accuracy: +1.4% | Helped |
| extend training schedule on the larger dataset | AdvXL ViT-L/16 on ImageNet-21K · ImageNet-21K | PGD-20 robust accuracy: 56.1 -> 58.3 | Helped |
| train AdvXL ViT-g/14 on DataComp-1B | AdvXL · ImageNet-1K val, AutoAttack l∞ (ϵ∞=4/255) | robust accuracy: +11.4% | Helped |
| train AdvXL ViT-g/14 on DataComp-1B | AdvXL · ImageNet-1K val, AutoAttack l2 (ϵ2=2) | robust accuracy: 70.4% | Helped |
| train AdvXL ViT-g/14 on DataComp-1B | AdvXL · ImageNet-1K val, AutoAttack l1 (ϵ1=75) | robust accuracy: 46.7% | Helped |
| replace full-resolution inputs with resizing to 112×112 during pre-training (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 54.5 -> 52.5 | Hurt |
| use random masking at 50% (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 54.5 -> 51.9 | Hurt |
| use block masking at 75% (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 54.5 -> 49.3 | Hurt |
| use PGD-2 rather than PGD-1 during pre-training (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 52.5 -> 52.6 | No clear effect |
| use PGD-3 rather than PGD-1 during pre-training (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 52.5 -> 52.5 | No clear effect |
| use PGD-3 rather than PGD-1 during fine-tuning (ablation) | AdvXL ViT-B/16 · ImageNet-1K | PGD-20 robust accuracy: 50.6 -> 52.5 | Helped |
Surprising findings
- Adversarially trained models generalize to unseen l2 and l1 attacks substantially better than prior l-inf-robust models, despite being trained only for l-inf robustness.
- Pre-training with weak PGD-1 attacks at reduced resolution matches the robustness of PGD-3 pre-training while cutting compute substantially.
Limitations
- Full-scale runs on billion-sample datasets require thousands of TPU/GPU-days, so results are hard to reproduce at scale.
- Swin-Transformer could not be trained with reduced-size inputs without architectural changes, so architecture comparison is limited to ViT and ConvNeXt.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K LAION-400M DataComp-1B
Benchmarks: ImageNet-1K
Metrics: robust accuracy under AutoAttack (l-inf, l2, l1) PGD-20 robust accuracy clean accuracy
Baselines: RobArch-L ViT-B/16 (Singh et al.) ConvNeXt-B+ConvStem ConvNeXt-L+ConvStem Swin-B Swin-L ConvNeXt-L
Builds on: PGD-based adversarial training CLIP CLIPA FastViT-style token reduction (masking/resizing)
Closest papers by meaning
- Bag of Tricks for Adversarial Training (ICLR 2021 · similarity 0.89)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.88)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.87)
- Explaining the role of Intrinsic Dimensionality in Adversarial Training (ICML 2025 · similarity 0.87)
- LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders (NeurIPS 2025 · similarity 0.87)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.