Research map / Adversarial robustness and security / Certified and empirical robustness
Average Certified Radius is a Poor Metric for Randomized Smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that average certified radius (ACR) is a poor metric for randomized smoothing, and that a Gaussian-training variant focusing on easy samples (discarding hard samples, reweighting, adaptive noise attack) matches SOTA ACR on CIFAR-10 while hurting hard samples.
Problem
Average certified radius (ACR) is the dominant metric for comparing randomized smoothing training methods, but its validity as a measure of true robustness under RS is unclear.
Approach
Theoretically shows ACR can be made arbitrarily large by a trivial classifier and is dominated by easy samples (p_A near 1); empirically shows SOTA RS training reduces accuracy on hard samples; proposes simple easy-sample-focused modifications to Gaussian training that match SOTA ACR and recommends the empirical distribution of p_A as an alternative metric.
Key results
- On CIFAR-10 with sigma=0.5, the proposed method reaches ACR 0.760 versus 0.757 for CAT-RS and 0.525 for Gaussian (ablation table), and 0.842 vs 0.846 for SmoothMix on ImageNet at sigma=0.5 in the main table.
- A trivial classifier can have arbitrarily large ACR given enough certification budget; increasing p_A from 0.99 to 0.999 raises the certified radius from 2.3 to 3.0 at N=10^5, matching the gain from p_A 0 to 0.76.
- Existing SOTA RS training methods put relatively more gradient weight on easy samples (easy/hard ratio 4.28 for CAT-RS vs 0.45 for Gaussian) and have lower density of p_A at moderate values than Gaussian training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed easy-sample-focused training (hard-data discarding, radius-based reweighting, and adaptive attack) instead of CAT-RS | Randomized smoothing on CIFAR-10 · CIFAR-10, σ=0.25 | ACR: 0.562 -> 0.564 | Helped |
| Use the proposed easy-sample-focused training instead of CAT-RS | Randomized smoothing on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.757 -> 0.760 | Helped |
| Use the proposed easy-sample-focused training instead of CAT-RS | Randomized smoothing on CIFAR-10 · CIFAR-10, σ=1.0 | ACR: 0.815 -> 0.844 | Helped |
| Use the proposed training instead of SmoothAdv | Randomized smoothing on ImageNet · ImageNet, σ=0.25 | ACR: 0.532 -> 0.529 | Hurt |
| Use the proposed training instead of SmoothMix | Randomized smoothing on ImageNet · ImageNet, σ=0.5 | ACR: 0.846 -> 0.842 | Hurt |
| Use the proposed training instead of CAT-RS | Randomized smoothing on ImageNet · ImageNet, σ=1.0 | ACR: 1.071 -> 1.042 | Hurt |
| Add hard-data discarding to Gaussian training (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.627 | Helped |
| Add dataset reweighting by approximate certified radius to Gaussian training (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.662 | Helped |
| Add adaptive adversarial noise selection to Gaussian training (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.701 | Helped |
| Combine hard-data discarding and dataset weighting (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.672 | Helped |
| Combine hard-data discarding and adaptive attack (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.731 | Helped |
| Combine dataset weighting and adaptive attack (ablation) | Gaussian training on CIFAR-10 · CIFAR-10, σ=0.5 | ACR: 0.525 -> 0.741 | Helped |
Surprising findings
- A trivial classifier that predicts one class can achieve ACR larger than a classifier with p_A=0.9 on every input for large enough budgets.
- Ranking of models by ACR reverses as the certification budget N changes in a toy binary example.
Limitations
- Experiments are limited to CIFAR-10 and ImageNet with a single certification algorithm (Cohen et al. style Certify) and the L2 setting.
- The proposed method gives lower certified accuracy than Gaussian training at small radii and does not improve robustness on the general data distribution.
- The ECDF-based alternative metric relies on a large N and assumes the ECDF is accurate.
Experimental setup
Datasets: CIFAR-10 ImageNet
Benchmarks: CIFAR-10 ImageNet
Metrics: average certified radius certified accuracy at radii empirical distribution of p_A (ECDF) gradient norm
Baselines: Gaussian SmoothAdv MACER Consistency SmoothMix CAT-RS
Builds on: randomized smoothing Gaussian training CAT-RS PGD
Closest papers by meaning
- Effects of Exponential Gaussian Distribution on (Double Sampling) Randomized Smoothing (ICML 2024 · similarity 0.91)
- Treatment of Statistical Estimation Problems in Randomized Smoothing for Adversarial Robustness (NeurIPS 2024 · similarity 0.90)
- Certified Adversarial Robustness via Randomized $\alpha$-Smoothing for Regression Models (NeurIPS 2024 · similarity 0.89)
- Mitigating the Curse of Dimensionality for Certified Robustness via Dual Randomized Smoothing (ICLR 2024 · similarity 0.88)
- The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing (ICLR 2024 · similarity 0.88)
- DRF: Improving Certified Robustness via Distributional Robustness Framework (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.