Research map / Adversarial robustness and security / Certified and empirical robustness
Clarifying the Behavior and the Difficulty of Adversarial Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an approximate analytic dynamics of recursive multi-step adversarial perturbations on a two-layer ReLU network, and uses it to explain why adversarial training is hard to optimize (imbalanced sample influence and parameter oscillation).
Problem
Adversarial training is considerably harder to optimize than vanilla training, and the reasons are not well understood; the paper asks how multi-step adversarial perturbations behave and why this makes training difficult.
Approach
Approximates the dynamics of a multi-step (PGD-style) attack in closed form on a two-layer ReLU network under three simplifying assumptions (plain gradient ascent, fixed gating states, infinite-step limit), showing perturbations amplify top Hessian eigen-directions exponentially and that adversarial training over-weights samples with large gradient norms and strengthens Hessian influence on parame
Key results
- Theoretical perturbation matches real PGD perturbations on 10 MLP/CNN/ResCNN architectures with relative error at about 1e-8 to 1e-4 (Table 1).
- Adversarial training makes samples with large gradient norms have roughly exp(A) times larger influence than vanilla training (conclusion C.5), supported by Figure 3 and Figure 4 on MNIST.
- Instability of weight gradients is measured to be higher under adversarial training than vanilla training for AlexNet and VGG-11 on MNIST (Figure 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-step adversarial perturbations instead of the original input during training | Adversarial training · Adversarial training (theoretical analysis) | influence of high-gradient-norm samples | Mixed |
| Increase the overall adversarial strength β | Multi-step adversarial attack · Adversarial perturbation generation | perturbation norm | Hurt |
| Increase the overall adversarial strength β | Multi-step adversarial attack · Adversarial perturbation generation | gradient norm | Hurt |
| Normalize or regularize perturbations in ℓ2 or ℓ∞ attacks | Adversarial training · Adversarial training | imbalance across input samples | Helped |
| Train on adversarial examples rather than original inputs | Adversarial training · Adversarial training | network-parameter oscillation | Hurt |
| Train on adversarial examples rather than original inputs | Adversarial training · Adversarial training | influence of samples with large Hessian values and gradient norms | Hurt |
Surprising findings
- Adversarial perturbations do not significantly change ReLU gating states, so the two-layer analysis transfers to deeper networks despite the simplifying assumption.
- l2 and l-infinity attacks reduce the imbalance across samples relative to the plain gradient attack, though the imbalance remains larger than in vanilla training.
Limitations
- Theory is derived for a two-layer ReLU network under three simplifying assumptions (gradient ascent without normalization, fixed gating states, infinite-step limit).
- Experiments are limited to MNIST and CIFAR-10 with small and medium networks.
- Oscillation claim relies on an optimization claim and is not directly proven that adversarial training increases the largest Hessian eigenvalue.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: relative error kappa between theoretical and real perturbati matching error 1-cos gradient and perturbation norm growth instability of weight gradients cosine similarity of sample effects
Baselines: vanilla training l2 PGD attack l-infinity PGD attack
Builds on: PGD adversarial training adversarial training min-max formulation gradient masking analysis
Closest papers by meaning
- A Closer Look at Curriculum Adversarial Training: From an Online Perspective (AAAI 2024 · similarity 0.91)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.90)
- Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality (NeurIPS 2020 · similarity 0.90)
- Bag of Tricks for Adversarial Training (ICLR 2021 · similarity 0.90)
- Taxonomy Driven Fast Adversarial Training (AAAI 2024 · similarity 0.88)
- A Sublinear Adversarial Training Algorithm (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.