Research map / Learning theory and algorithms / Sample complexity
Uniformly Stable Algorithms for Adversarial Training and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ME-A, a Moreau-envelope-based wrapper around any first-order algorithm that achieves uniform stability for weakly-convex non-smooth losses, and applies it to adversarial training to mitigate robust overfitting.
Problem
Robust overfitting in adversarial training is linked to the non-smoothness of the adversarial loss, which makes SGD non-uniformly stable; existing uniformly stable methods need convexity and are computationally impractical.
Approach
Apply the Moreau envelope to the empirical loss (not per-sample) to get a min-min problem; alternate a first-order algorithm on the strongly convex inner problem with gradient descent on the convex smooth outer problem, giving uniform stability for weakly-convex non-smooth losses at no extra overhead.
Key results
- ME-SGD with TRADES loss reaches 55.23% AutoAttack robust accuracy on CIFAR-10 vs 51.91% for SGD; with extra data 62.76% vs 59.45%.
- With adversarial (AT) loss, ME-SGD reaches 51.66% vs 50.80% for SGD on CIFAR-10 under AutoAttack.
- ME-A mitigates robust overfitting: robust test accuracy no longer declines after ~100 epochs on SVHN, CIFAR-10 and CIFAR-100, unlike SGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ME-𝒜 with SGD instead of SGD | Adversarial training on CIFAR-10 with WideResNet-28×10 · CIFAR-10 AutoAttack, AT Loss | robust accuracy: 50.80{ ±0.23%} -> 51.66{ ±0.16%} | Helped |
| Apply ME-𝒜 with SGD instead of SGD | Adversarial training on CIFAR-10 with WideResNet-28×10 · CIFAR-10 AutoAttack, TRADES | robust accuracy: 51.91% -> 55.23{ ±0.19%} | Helped |
| Add unlabeled data at labeled-to-unlabeled ratio 3:7 | SGD and ME-SGD adversarial training on CIFAR-10 · CIFAR-10 AutoAttack, AT Loss | robust accuracy: 50.80{ ±0.23%} -> 58.41{ ±0.25%}; 51.66{ ±0.16%} -> 59.14{ ±0.18%} | Helped |
| Add unlabeled data at labeled-to-unlabeled ratio 3:7 | SGD and ME-SGD adversarial training on CIFAR-10 · CIFAR-10 AutoAttack, TRADES | robust accuracy: 51.91% -> 59.45%; 55.23{ ±0.19%} -> 62.76{ ±0.15%} | Helped |
| Vary p from 0 to 1×10⁵ in ME-𝒜 (ablation) | ME-𝒜 on CIFAR-10 with TRADES loss · CIFAR-10 TRADES | robust accuracy: 63.17 -> 63.86; 63.41 | No clear effect |
Surprising findings
- Stochastic weight averaging (the p->0 limit of ME-A) performs comparably to ME-A in practice, though only ME-A has provable guarantees.
- The Moreau parameter p has little effect on robust accuracy across a wide range (0 to 1e5), with best result at p=1.
Limitations
- Theory relies on weak convexity of the adversarial loss, which is a gap from the non-convex nature of deep networks.
- Inner problem needs O(n^2) iterations for uniform stability guarantees in the non-smooth case.
- Experiments focus on A=SGD and a limited set of datasets and architectures.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN pseudo-labeled data (from prior work)
Benchmarks: CIFAR-10 CIFAR-100 SVHN AutoAttack
Metrics: robust test accuracy generalization gap robust test error
Baselines: SGD adversarial training SWA PPM ERM (weight decay) TRADES
Builds on: Moreau envelope proximal point method Moreau-Yosida smoothing PGD adversarial training TRADES stochastic weight averaging
Closest papers by meaning
- Stability Analysis and Generalization Bounds of Adversarial Training (NeurIPS 2022 · similarity 0.91)
- On the Algorithmic Stability of Adversarial Training (NeurIPS 2021 · similarity 0.90)
- Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation (NeurIPS 2024 · similarity 0.88)
- Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise (AAAI 2024 · similarity 0.87)
- On the Stability and Generalization of Meta-Learning (NeurIPS 2024 · similarity 0.86)
- Adversarial Training Should Be Cast as a Non-Zero-Sum Game (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.