Research map / Adversarial robustness and security / Certified and empirical robustness
AttackBench: Evaluating Gradient-based Attacks for Adversarial Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttackBench, a query-budgeted benchmark that ranks over 100 gradient-based adversarial attack implementations across CIFAR-10 and ImageNet models using a new local/global optimality metric based on robustness evaluation curves.
Problem
Gradient-based adversarial attacks are compared under inconsistent setups (models, metrics, implementations, and query budgets), leading to biased and unfair evaluations.
Approach
An evaluation framework that wraps target models to cap forward/backward queries, records best-found adversarial examples, and scores attacks by "optimality": area-under-robustness-curve gap relative to an empirical best attack formed by per-sample minimum over all tested attacks.
Key results
- Sigma-zero and DDN are top-ranked for l0 and l2 norms, achieving GO near 1 (e.g., sigma-zero l0 GO 0.984 on CIFAR-10).
- APGD-l1, APGD-t and PDPGD consistently appear in top-5 for l1 and linf norms across CIFAR-10 and ImageNet.
- Implementation differences matter: APGD in Art scores GO 0.26 under l1 on CIFAR-10 versus 0.909 for the original implementation, due to restart count (5 vs 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Art implementation instead of the original implementation of APGD-ℓ1 | APGD-ℓ1 · CIFAR-10 ℓ1 | GO: 90.9% -> 26% | Hurt |
| Set APGD random initializations to 5 rather than 1 | APGD · CIFAR-10 ℓ1 | GO: 90.9% -> 26% | Hurt |
| Use AdvLib APGD instead of original APGD | APGD · ImageNet ℓ2 | GO: 36.2% -> 95.9% | Helped |
| Use AdvLib APGD instead of original APGD | APGD · ImageNet ℓ∞ | GO: 39.1% -> 97% | Helped |
| Use DLR loss rather than NCE loss in APGD | APGD · ImageNet ℓ2 and ℓ∞ | GO: 36.2% -> 95.9%; 39.1% -> 97% | Helped |
| Use APGD rather than PDPGD on ImageNet ℓ1 | APGD vs PDPGD · ImageNet ℓ1 | GO: 0.973 -> 0.980 | Mixed |
| Scale VFGA from CIFAR-10 to ImageNet | VFGA · ℓ0 | GO: 0.802 -> 0.712 | Hurt |
| Use a dynamically reducing step size rather than a fixed step size | Gradient-based attacks · CIFAR-10 and ImageNet attack benchmark | GO | Helped |
Surprising findings
- Different library implementations of the same attack can differ substantially in optimality (e.g., APGD l1 on CIFAR-10: 90.9% original vs 26% in Art).
- No evaluated attack reaches 100% global optimality, and several widely used attacks rank among the worst performers.
Limitations
- Restricted to gradient-based attacks on image classifiers; black-box attacks and other domains not covered.
- Optimality is relative to the empirical best over the tested attacks, not the true optimum.
- Runtime comparisons depend on hardware.
Experimental setup
Datasets: CIFAR-10 ImageNet
Benchmarks: CIFAR-10 ImageNet
Metrics: attack success rate local optimality global optimality forward and backward query counts runtime
Baselines: APGD APGD-t FMN DDN sigma-zero PDPGD VFGA PGD-l0 BIM DeepFool
Builds on: PGD APGD FMN DDN Foolbox AdvLib
Closest papers by meaning
- Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples (NeurIPS 2022 · similarity 0.90)
- RobustBench: a standardized adversarial robustness benchmark (NeurIPS 2021 · similarity 0.89)
- $\sigma$-zero: Gradient-based Optimization of $\ell_0$-norm Adversarial Examples (ICLR 2025 · similarity 0.87)
- TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases (NeurIPS 2024 · similarity 0.87)
- Improved Generation of Adversarial Examples Against Safety-aligned LLMs (NeurIPS 2024 · similarity 0.87)
- SuperDeepFool: a new fast and accurate minimal adversarial attack (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.