Research map / Adversarial robustness and security / Certified and empirical robustness
Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AR-AT, an asymmetric representation-level invariance regularizer (stop-gradient plus predictor MLP) combined with split-BatchNorm to improve the robustness-accuracy trade-off in adversarial training.
Problem
Adversarial training suffers a robustness-accuracy trade-off; invariance regularization still loses clean accuracy due to gradient conflict between invariance and classification losses and a BatchNorm mixture-distribution problem between clean and adversarial inputs.
Approach
Asymmetrize the invariance loss with a stop-gradient and a predictor MLP so only the adversarial representation is pulled toward the clean one, and use split-BN (separate auxiliary BNs for clean inputs during training only) to avoid mixed batch statistics; also offers a view that KD-based defenses succeed by resolving these same two issues.
Key results
- On CIFAR-10 with WRN-34-10, AR-AT reaches 90.89% clean and 50.77% AutoAttack accuracy (sum 141.66), above LBGAT (140.75) and ARREST (140.44); AR-AT+SWA reaches sum 144.09.
- On CIFAR-100 with WRN-34-10, AR-AT+SWA attains a sum of 99.69 (AA 27.31), versus LBGAT at 95.09.
- Naive invariance regularization shows over 40% of parameters with gradient conflict; stop-gradient alone or split-BN alone gives limited gains, while combining both raises ResNet-18 CIFAR-10 AA from 46.50 to 49.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AR-AT instead of LBGAT | ResNet-18 · CIFAR10 | Clean accuracy: 85.00 ± 0.47 -> 87.82 ± 0.19 | Helped |
| Add SWA to AR-AT | WRN-34-10 · CIFAR10 | AutoAttack accuracy: 50.77 ± 0.50 -> 54.03 ± 0.31 | Helped |
| Add stop-gradient to naive invariance regularization (ablation) | AR-AT, ResNet-18 · CIFAR10 | Clean accuracy: 82.93 -> 82.47 | Hurt |
| Add stop-gradient to naive invariance regularization (ablation) | AR-AT, ResNet-18 · CIFAR10 | PGD-20 accuracy: 48.89 -> 48.57 | Hurt |
| Add split-BN to naive invariance regularization (ablation) | AR-AT, ResNet-18 · CIFAR10 | PGD-20 accuracy: 48.89 -> 51.34 | Helped |
| Add predictor MLP to stop-gradient plus split-BN (ablation) | AR-AT, ResNet-18 · CIFAR10 | PGD-20 accuracy: 51.46 -> 52.40 | Helped |
| Use auto-balance instead of the best listed fixed setting (ablation) | AR-AT, WRN-34-10 · CIFAR10 | PGD-20 accuracy: 50.35 -> 52.72 | Helped |
| Regularize all ReLUs in layer4 instead of the penultimate layer (ablation) | AR-AT, ResNet-18 · CIFAR10 | Clean accuracy: 86.28 -> 87.93 | Helped |
| Use auxiliary clean BNs at inference instead of main BNs (ablation) | AR-AT, ResNet-18 · CIFAR10 | PGD-20 accuracy: 52.13 -> 1.98 | Hurt |
| Asymmetrize TRADES with stop-gradient and split-BN (ablation) | TRADES, ResNet-18 · CIFAR10 | Clean accuracy: 81.25 -> 85.62 | Helped |
| Asymmetrize TRADES with stop-gradient and split-BN (ablation) | TRADES, ResNet-18 · CIFAR10 | AutoAttack accuracy: 48.54 -> 48.58 | No clear effect |
Surprising findings
- Stop-gradient alone does not reliably improve the trade-off and can increase the adversarial-clean feature distance, exacerbating the mixture distribution problem.
- Asymmetrized TRADES with stop-gradient and split-BN achieves roughly the same performance as LBGAT, suggesting KD-based gains stem from resolving these issues.
Limitations
- Determining which layers to regularize remains ambiguous for novel architectures.
- Not directly applicable to models without BatchNorm such as Transformers using LayerNorm.
- Code promised only upon publication; ARREST results are copied from the original paper rather than reproduced.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Imagenette
Benchmarks: CIFAR-10 CIFAR-100 Imagenette AutoAttack PGD-20
Metrics: clean accuracy robust accuracy (AutoAttack) sum of clean and robust accuracy gradient similarity cosine similarity of adversarial and clean features
Baselines: AT TRADES MART LBGAT ARREST AWP TRADES-AWP AT+HF
Builds on: TRADES LBGAT SimSiam MBN-AT AdvProp
Closest papers by meaning
- Consistency Regularization for Adversarial Robustness (AAAI 2022 · similarity 0.91)
- Towards Understanding the Regularization of Adversarial Robustness on Neural Networks (ICML 2020 · similarity 0.90)
- Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization (NeurIPS 2025 · similarity 0.90)
- Towards Better Robust Generalization with Shift Consistency Regularization (ICML 2021 · similarity 0.90)
- Why Adversarially Train Diffusion Models? (ICLR 2026 · similarity 0.90)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.