Research map / Adversarial robustness and security / Certified and empirical robustness
Robustness Tokens: Towards Adversarial Robustness of Transformers
Brian Pulfer, Yury Belousov, Slava Voloshynovskiy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| append 10 learned Robustness Tokens to DiNOv2 input | DiNOv2 (S/B/L/G) · ImageNet-1k linear classification (clean) | accuracy: average -0.5 | Hurt |
| append 10 learned Robustness Tokens to DiNOv2 input | DiNOv2 (S/B/L/G) · ADE20K linear segmentation (clean) | mIoU: +0.2 | No clear effect |
| add Robustness Tokens | DiNOv2-B · ImageNet PGD attack classification | accuracy: 0.0 -> 50.0 | Improved |
| add Robustness Tokens | DiNOv2-B · ADE20K PGD attack segmentation | mIoU: 3.1 -> 23.4 | Improved |
| add Robustness Tokens to larger models | DiNOv2-S vs G · ImageNet PGD attack classification | accuracy: 31.9 (S) vs 63.1 (G) | Improved |
| increase number of Robustness Tokens beyond 10 (1, 10, 20, 50) (ablation) | DiNOv2 · training loss | training loss | No clear effect |
| use a single Robustness Token (ablation) | DiNOv2 · feature orientation | training loss | Improved |
| tokens trained on PGD, tested with AutoAttack | DiNOv2-L · ImageNet val AutoAttack | accuracy: 1.8 -> 60.1 | Improved |
| add Robustness Tokens to OpenCLIP | OpenCLIP Base · clean-vs-adversarial cosine similarity | avg cosine similarity: -0.02 -> 0.86 | Improved |
| add Robustness Tokens to DEIT-III | DEIT-III Base · clean-vs-adversarial cosine similarity | avg cosine similarity: 0.16 -> 0.74 | Improved |
| train tokens for about 200 steps with batch size 8 | Robustness Tokens · ImageNet subset (1600 images) | training convergence: converges in under 200 steps | Improved |
Closest papers by meaning
- Adversarially Pretrained Transformers may be Universally Robust In-Context Learners (ICLR 2026 · similarity 0.90)
- Are Transformers more robust than CNNs? (NeurIPS 2021 · similarity 0.90)
- GaLileo: General Linear Relaxation Framework for Tightening Robustness Certification of Transformers (AAAI 2024 · similarity 0.88)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.88)
- ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · similarity 0.88)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
