Research map / Adversarial robustness and security / Certified and empirical robustness
Boosting Adversarial Training via Fisher-Rao Norm-based Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOAT, an epoch-dependent logit-based regularizer motivated by Fisher-Rao norm Rademacher complexity bounds, which boosts standard accuracy of several adversarial training methods on CIFAR10 with little overhead.
Problem
Adversarial training degrades standard (clean) generalization, the robustness-accuracy trade-off, and the paper asks whether model complexity can explain and mitigate this degradation.
Approach
Uses Fisher-Rao norm to bound the cross-entropy-loss Rademacher complexity of ReLU MLPs, identifies a logit-based variable Γce that correlates with the clean CE generalization gap (positively early, negatively late in training), and builds an epoch-specific regularizer (LOAT) combining logit-oriented penalties with adaptive adversarial logit pairing.
Key results
- MART+LORE improves ResNet-18 clean accuracy by 0.92% over MART (88.07/89.04 range reported in ablation; Tab.1 shows 89.04 vs 87.85 train-clean baseline).
- With EDM-augmented CIFAR10, TRADES+LORE raises clean accuracy by 2.29% on PreResNet18 (Swish) and 1.80% on WRN-28-10 (Swish).
- TRADES's FGSM accuracy on ResNet-18 improves by 4.58% with LOAT, and TRADES (S)'s by 4.62%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SLORE to PGD-AT | ResNet18 adversarial training · CIFAR10 | PGD20 accuracy: 44.94 -> 46.50 | Helped |
| Add LORE to MART | ResNet18 adversarial training · CIFAR10 | clean test accuracy: 80.61 -> 81.53 | Helped |
| Add LORE to TRADES | ResNet18 adversarial training · CIFAR10 | PGD20 accuracy: 52.00 -> 50.43 | Hurt |
| Add LORE to TRADES | PreResNet18 (Swish) adversarial training · CIFAR10 EDM-augmented | clean test accuracy: 86.17 -> 88.46 | Helped |
| Add LORE to TRADES (S) | PreResNet18 (Swish) adversarial training · CIFAR10 EDM-augmented | AutoAttack accuracy: 54.61 -> 55.06 | Helped |
| Add LORE to MART | WRN-28-10 (Swish) adversarial training · CIFAR10 EDM-augmented | PGD40 accuracy: 62.91 -> 66.03 | Helped |
| Apply SLORE before E1 and after E2 rather than at only one phase (ablation) | ResNet18 PGD-AT · CIFAR10 | clean test accuracy: 83.39 -> 83.41; 82.65 -> 83.41 | Helped |
| Apply LORE before E1 and after E2 rather than at only one phase (ablation) | ResNet18 PGD-AT · CIFAR10 | clean test accuracy: 83.07 -> 83.25; 82.67 -> 83.25 | Helped |
| Use both P̌C and P̌M instead of P̌C alone (ablation) | ResNet18 PGD-AT with SLORE · CIFAR10 | clean test accuracy: 83.01 -> 83.41 | Helped |
| Use both P̌C and P̌M instead of P̌M alone (ablation) | ResNet18 PGD-AT with SLORE · CIFAR10 | clean test accuracy: 82.92 -> 83.41 | Helped |
| Use both P̌C and P̌M instead of P̌C alone (ablation) | ResNet18 PGD-AT with LORE · CIFAR10 | clean test accuracy: 82.80 -> 83.25 | Helped |
| Use both P̌C and P̌M instead of P̌M alone (ablation) | ResNet18 PGD-AT with LORE · CIFAR10 | clean test accuracy: 82.72 -> 83.25 | Helped |
Surprising findings
- The correlation between Γce and the CE generalization gap flips sign between early (10 epochs) and late (1000 epochs) training, motivating an epoch-dependent regularizer.
Limitations
- Theoretical analysis is for ReLU MLPs; extension to deep CNNs is only empirical via the regularizer.
- Gains are modest in several settings and some metrics (e.g., PGD-20 for TRADES) decline slightly.
Experimental setup
Datasets: CIFAR10 EDM-generated CIFAR10 images
Benchmarks: CIFAR10
Metrics: clean accuracy robust accuracy under FGSM PGD-7 PGD-20 PGD-40 AutoAttack
Baselines: PGD-AT TRADES TRADES (LSE) MART DM-AT
Builds on: PGD-AT TRADES TRADES (LSE) MART Adversarial Logit Pairing
Closest papers by meaning
- Adversarial Training is a Form of Data-dependent Operator Norm Regularization (NeurIPS 2020 · similarity 0.89)
- Consistency Regularization for Adversarial Robustness (AAAI 2022 · similarity 0.89)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.88)
- Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality (NeurIPS 2020 · similarity 0.88)
- Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization (NeurIPS 2025 · similarity 0.86)
- Efficient local linearity regularization to overcome catastrophic overfitting (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.