Research map / Adversarial robustness and security / Certified and empirical robustness
Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives uniform-argument-stability and robust generalization bounds for gradient-descent and SGD adversarial training of two-layer smooth-activation networks with logistic loss, without data distribution assumptions, and shows Moreau-envelope smoothing improves stability.
Problem
Understanding why and when adversarial training generalizes, beyond the NTK regime and without restrictive data assumptions such as linear separability or robust realizability.
Approach
Analyzes uniform argument stability of adversarial training for over-parameterized two-layer networks by showing the robust empirical loss is approximately (weakly) convex in the weights, and smooths the robust loss via Moreau's envelope to remove the O(eta sqrt(T)) non-smoothness term.
Key results
- Theorem 3.1: GD adversarial training uniform argument stability bound O(eta sqrt(T) + eta T/n + sqrt(beta1 eta T)) for over-parameterized two-layer networks.
- Theorem 3.2: robust generalization gap and robust loss are bounded via the ARERM oracle with early stopping, given width m >= H^2 C_x^4 eta^2 (T+1)^2.
- Theorem 3.5: Moreau-envelope gradient descent achieves stability O(C_x eta T / n + eta T sqrt(beta2/mu)), removing the O(eta sqrt(T)) term present in standard adversarial training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Moreau smoothing instead of direct adversarial training | Two-layer neural network adversarial training · Robust adversarial training | uniform argument stability bound: O(ηT/n) vs. O(η√T + ηT/n) | Helped |
| Set adversarial-example precision β1 = 0 (ablation) | GD adversarial training · Robust adversarial training | uniform argument stability bound: O(η√T + ηT/n) | Helped |
| Set Moreau minimizer precision β2 = 0 (ablation) | GD on Moreau envelope · Robust adversarial training | uniform argument stability bound: O(ηT/n) | Helped |
| Use early stopping with sufficiently large network width | GD adversarial training · Robust adversarial training | robust loss bound: O(1/√T) | Helped |
| Use a good initialization near a robust network with small robust loss | GD adversarial training · Robust adversarial training | robust loss: O(1/√T) | Helped |
| Use Moreau smoothing | GD adversarial training · Robust adversarial training | robust generalization | Mixed |
Limitations
- Results are for two-layer networks with smooth activations only; extension to deep networks is open.
- Moreau envelope method requires solving a strongly convex min-max problem at each step, increasing computational cost.
- Bounds are in expectation and require sufficiently large width m; minimum over last iterates rather than last iterate in general.
Experimental setup
Metrics: uniform argument stability robust generalization gap robust test loss
Baselines: standard adversarial training (GD) SGD adversarial training
Builds on: Hardt et al. 2016 stability of SGD Xing et al. 2021 adversarial training stability Xiao et al. 2024 Moreau envelope stability Richards and Kuzborskij 2021 Madry et al. PGD adversarial training
Closest papers by meaning
- Stability Analysis and Generalization Bounds of Adversarial Training (NeurIPS 2022 · similarity 0.91)
- Algorithmic Stability Based Generalization Bounds for Adversarial Training (ICLR 2025 · similarity 0.90)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.89)
- Uniformly Stable Algorithms for Adversarial Training and Beyond (ICML 2024 · similarity 0.88)
- Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach (ICLR 2024 · similarity 0.88)
- Adversarial Training Should Be Cast as a Non-Zero-Sum Game (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.