Research map / Adversarial robustness and security / Certified and empirical robustness

Robust NAS under adversarial training: benchmark, theory, and beyond

ICLR 2024 · Adversarial robustness and security › Certified and empirical robustness · approach: Robustness-accuracy trade-off analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Releases NAS-RobBench-201, an adversarially trained benchmark of 6,466 NAS-Bench-201 architectures on CIFAR-10/100 and ImageNet-16-120, and derives NTK-based generalization bounds for clean and robust accuracy under multi-objective adversarial training.

Problem

Existing NAS benchmarks evaluate architectures only under standard training, so robust architecture search under adversarial training lacks reproducible tabular evaluations and theoretical guarantees on generalization for clean and robust accuracy.

Approach

Builds a look-up benchmark by adversarially training every non-isomorphic NAS-Bench-201 cell and extends NTK generalization theory to a multi-objective loss mixing clean and PGD-adversarial terms, showing clean accuracy depends on a joint clean/robust NTK and robust accuracy on a robust NTK and its "twice"-perturbed version.

Key results

• Benchmark covers 6,466 non-isomorphic architectures trained adversarially on three datasets with three seeds, costing about 107k GPU hours.
• Local search finds an architecture with clean 0.796 and PGD(8/255) 0.478 on CIFAR-10, close to the optimal 0.794/0.482 found by exhaustive benchmark lookup.
• Clean and robust rankings correlate strongly on the benchmark (Spearman about 0.97-0.99 among clean, PGD and FGSM metrics), while the correlation with the standard-training benchmark's PGD ranking is weak (about 0.31-0.38).

Measured effects reported in the paper

Surprising findings

• Architecture rankings under standard-training robustness (NAS-Bench-201 robustness benchmark) correlate only weakly (about 0.31-0.38 Spearman) with rankings under adversarial training on the PGD metric.
• Search using robust metrics (FGSM/PGD) finds more robust architectures than searching with clean accuracy, but the gap across search algorithms is small.

Limitations

• Benchmark and theory do not cover modern Vision Transformers.
• NTK analysis is restricted to the linear (lazy) training regime and may not explain practical success of deep networks.
• Lower bounds on minimum NTK eigenvalue are estimated via Frobenius norm proxies, so the NTK-score correlations are approximate.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet-16-120

Benchmarks: CIFAR-10 CIFAR-100 ImageNet-16-120

Metrics: clean accuracy robust accuracy under FGSM robust accuracy under PGD (3/255, 8/255) AutoAttack robust accuracy Spearman correlation

Baselines: Regularized Evolution Random Search Local Search NAS-Bench-201 standard-training robustness benchmark

Builds on: NAS-Bench-201 neural tangent kernel PGD adversarial training Fast gradient sign method

Closest papers by meaning

• Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations (NeurIPS 2023 · similarity 0.93)
• Towards Accurate and Robust Architectures via Neural Architecture Search (CVPR 2024 · similarity 0.93)
• When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks (CVPR 2020 · similarity 0.93)
• Robustifying and Boosting Training-Free Neural Architecture Search (ICLR 2024 · similarity 0.89)
• First Line of Defense: A Robust First Layer Mitigates Adversarial Attacks (AAAI 2025 · similarity 0.89)
• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.89)