Research map / Adversarial robustness and security / Black-box and transfer attacks
Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Probability Margin Attack (PMA), an individual white-box attack whose margin loss is defined in probability space, plus PMA-based ensembles and a million-scale CC1M dataset for evaluating adversarially trained ImageNet models.
Problem
White-box adversarial robustness evaluation needs attacks that are both strong and efficient, especially for large-scale testing where ensemble attacks like AutoAttack are too costly and small test sets may not reflect real-world risk.
Approach
Defines the adversarial margin in probability space (p_max - p_y) instead of logit space, shown to be a weighted combination of targeted and untargeted cross-entropy gradients, and plugs it into the Margin Decomposition attack pipeline; ensembles PMA with one other attack (PMA+) or cascades many attacks.
Key results
- PMA yields lower robust accuracy than prior individual attacks across 29 RobustBench models, reducing robustness by up to 0.55% versus the MD attack.
- PMA+ (PMA + APGDT) outperforms AutoAttack on all CIFAR-10/100/ImageNet models in Table 6 while taking about 25% of AA's running time.
- Relative robustness of top-5 ImageNet robust models drops from above 59% on ImageNet-1k to below 19% on the 1M-image CC1M set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace existing individual attack losses with Probability Margin loss in the attack | PGD and APGD robustness evaluation · CIFAR-10, CIFAR-100, and ImageNet-1k | robust accuracy (%): 0.03% to 1.56% lower | Helped |
| Use PMA instead of the strongest individual attack baseline | Robustness evaluation · CIFAR-10, CIFAR-100, and ImageNet-1k | robust accuracy (%): up to 0.22% | Helped |
| Combine PMA with APGDT | PMA · CIFAR-10 | robust accuracy (%): 0.52% | Helped |
| Add more individual attacks sequentially to a PMA cascade ensemble | PMA cascade ensemble · CIFAR-10 | robust accuracy (%): 0.99% | Helped |
| Evaluate on CC1M rather than the ImageNet-1k test set | Adversarially trained ImageNet models · CC1M versus ImageNet-1k test set | relative robustness: above 59% to below 19% | Hurt |
| Use PMA instead of PGDce | Relative robustness evaluation · CC1M | relative robustness: 3.56% to 11.86% lower | Helped |
Surprising findings
- The relative robustness gap between small-scale ImageNet-1k testing and million-scale CC1M evaluation is large, and model rankings change.
Limitations
- PMA+ still requires considerable time for large-scale evaluation.
- Approach is limited to white-box settings and does not address black-box attacks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k CC3M
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k CC1M RobustBench leaderboard
Metrics: robust accuracy relative robustness running time
Baselines: PGD APGD AutoAttack Margin Decomposition attack FAB DLR loss margin loss MIFPE loss
Builds on: Margin Decomposition attack AutoAttack PGD DLR loss
Closest papers by meaning
- Benchmarking Adversarial Robustness on Image Classification (CVPR 2020 · similarity 0.91)
- Fantastic Robustness Measures: The Secrets of Robust Generalization (NeurIPS 2023 · similarity 0.91)
- RobustBench: a standardized adversarial robustness benchmark (NeurIPS 2021 · similarity 0.90)
- GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models (NeurIPS 2024 · similarity 0.89)
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.88)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.