Research map / Adversarial robustness and security / Certified and empirical robustness
Accelerating Adversarially Robust Model Selection for Deep Neural Networks via Racing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a racing-based model selection method with a Δ-value sorting mechanism to find the most adversarially robust neural network from a set of models using far less formal verification compute.
Problem
Selecting the most robust neural network model (highest certified robust accuracy) from a set of candidates is prohibitively expensive because formal local robustness verification takes hours per input.
Approach
Combine a racing algorithm (naive racing and an F-Race adaptation with Cochran's Q / McNemar and Dunn's post-hoc tests) with a sorting of inputs by a Δ-value (gap between top-two output logits), so that the least-robust-looking inputs are verified first and candidates are eliminated quickly.
Key results
- On 31 MNIST networks, the racing method with sorting achieved a 108-fold speedup over exhaustive evaluation (1380.93 vs 12.83 hours) while reaching optimal regret.
- On 27 CIFAR-10 networks, the method achieved a 42-fold speedup (558.44 vs 13.18 hours; reported as 41-fold in text) with optimal regret.
- Statistically significant (Kolmogorov-Smirnov, p<0.05) difference in Δ-value distributions between sat and unsat instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use naïve racing with Δ-value-sorted inputs instead of exhaustive evaluation | Robust model selection across MNIST networks · MNIST test set (100 instances) | cumulative running time: 1380.93 vs 12.83 hours; speedup factor of 108 | Helped |
| Use naïve racing with Δ-value-sorted inputs instead of exhaustive evaluation | Robust model selection across CIFAR-10 networks · CIFAR-10 test set (100 instances) | cumulative running time: 558.44 vs 13.18 hours; 41-fold speedup | Helped |
| Use naïve racing with Δ-value-sorted inputs instead of naïve racing without sorting | Robust model selection across CIFAR-10 networks · CIFAR-10 | regret | Helped |
Surprising findings
- For 12 of 31 MNIST and 15 of 27 CIFAR networks, some sat instances had large Δ-values, indicating Δ does not perfectly separate non-robust inputs when a model is generally weak.
Limitations
- Evaluated only on two datasets (MNIST, CIFAR-10) with 100 test instances each and fixed epsilon values.
- Δ-values only weakly reflect distance to decision boundary; non-robust models can have large Δ on sat instances.
- Relies on specific complete verifiers (VeriNet, beta-CROWN) and timeouts; results may not transfer to other verifiers.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST (31 ERAN networks, first 100 test instances) CIFAR-10 (27 ERAN networks, first 100 test instances)
Metrics: cumulative running time regret speedup factor number of candidate models
Baselines: Exhaustive evaluation F-Race Naive racing
Builds on: F-Race VeriNet beta-CROWN ERAN
Closest papers by meaning
- Evaluating Robustness of Neural Networks with Mixed Integer Programming (ICLR 2019 · similarity 0.89)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.88)
- Robust NAS under adversarial training: benchmark, theory, and beyond (ICLR 2024 · similarity 0.87)
- ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck (ICML 2025 · similarity 0.87)
- If You Want to Be Robust, Be Wary of Initialization (NeurIPS 2024 · similarity 0.86)
- On the Scalability of Certified Adversarial Robustness with Generated Data (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.