Research map / Adversarial robustness and security / Certified and empirical robustness

GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models

NeurIPS 2024 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GREAT Score, an attack-independent global robustness metric that averages a certified L2 lower bound on minimal perturbation over samples from an off-the-shelf generative model (GAN/diffusion), with a sample-complexity guarantee and efficient black-box evaluation.

Problem

Existing adversarial robustness evaluations aggregate local robustness on a finite set of test samples, which may be unrepresentative of the global robustness under the true data distribution; attack-based and certification methods are also costly and often white-box.

Approach

Define global robustness as the expectation of a certified L2 lower bound on minimal perturbation over a generative model's sampling distribution, using a closed-form score built from the top-1 confidence margin scaled by sqrt(pi/2) (via Stein's lemma), so only forward passes on generated samples are needed.

Key results

• Calibrated GREAT Score vs RobustBench Spearman correlation of 0.8971 on CIFAR-10, versus 0.6618 uncalibrated.
• GREAT Score per-sample run-time is roughly 800-2000 times faster than AutoAttack on 500 generated CIFAR-10 images.
• On ImageNet, GREAT Score vs RobustBench Spearman correlation is 0.8, higher than AutoAttack vs RobustBench (0.6).

Measured effects reported in the paper

Surprising findings

• AutoAttack model rankings vary notably across random 10,000-sample subsets (correlation 0.907 +/- 0.0256 with full test set), showing local attack-based rankings are unstable.
• Rankings from AutoAttack at epsilon 0.3 vs 0.7 on CIFAR-10 have Spearman correlation only 0.9485.

Limitations

• Currently centered on L2-norm perturbations; extension to other Lp norms depends on Stein's lemma extension.
• Relies on the generative model producing valid samples of the conditioned class; label ambiguity is not addressed.
• Calibration requires CW attack results on generated samples, adding cost, and uncalibrated correlation with RobustBench is only moderate (0.6618).

Experimental setup

Datasets: CIFAR-10 ImageNet-1K CelebA-HQ CelebA

Benchmarks: RobustBench CIFAR-10 RobustBench ImageNet CelebA-HQ online gender APIs

Metrics: Spearman rank correlation robust accuracy CW distortion run-time per sample

Baselines: AutoAttack CW attack RobustBench accuracy Square attack

Builds on: CLEVER Stein's lemma AutoAttack RobustBench StudioGAN

Closest papers by meaning

• Fantastic Robustness Measures: The Secrets of Robust Generalization (NeurIPS 2023 · similarity 0.92)
• RobustBench: a standardized adversarial robustness benchmark (NeurIPS 2021 · similarity 0.90)
• Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks (CVPR 2025 · similarity 0.89)
• OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shift (ICML 2024 · similarity 0.89)
• Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.88)
• Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.88)