Research map / Adversarial robustness and security / Certified and empirical robustness
Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CURE, an adversarial training framework that selectively conserves, updates (via a Robust Gradient Prominence mask) and revises (stochastic momentum teacher) weights to improve the natural-robust trade-off and reduce robust overfitting.
Problem
Adversarial training degrades clean (standard) accuracy, creating a trade-off between standard and robust generalization, and suffers from robust overfitting.
Approach
Layer-wise analysis shows selectively updating some layers while freezing others preserves natural knowledge and improves learning; CURE operationalizes this with a gradient-based Robust Gradient Prominence score that masks the lowest-scoring fraction of gradients each step, plus a stochastic momentum revision model with a consistency loss.
Key results
- On WideResNet-34-10 CIFAR-10, CURE achieves NRR 67.60 (Nat 87.05, C&W 55.25), the highest among compared methods including TRADES (66.17) and LAS-AT (67.19).
- On ResNet-18 SVHN, CURE reaches 92.77 natural and 57.34 PGD100 accuracy, with NRR 70.87 versus 66.69 for ST-AT.
- CURE shows zero best-to-last robust overfitting gap (Delta = 0.00) on CIFAR-10 for both PreActResNet-18 and WideResNet-34-10, where AT shows -5.75 and -5.20 AA drops.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RGP-based selective weight updates and Revision to adversarial training (ablation) | CURE on CIFAR-10, ResNet-18 · CIFAR-10 | NRR: 60.95 -> 63.19 | Helped |
| Add RGP-based selective weight updates (ablation) | CURE on CIFAR-10, ResNet-18 without Revision · CIFAR-10 | NRR: 60.95 -> 61.18 | Helped |
| Add Revision (ablation) | CURE on CIFAR-10, ResNet-18 without RGP · CIFAR-10 | NRR: 60.95 -> 60.37 | Hurt |
| Replace natural-data pretraining with a short natural-training warm-up (ablation) | CURE on CIFAR-10, ResNet-18 · CIFAR-10 | NRR: 65.04 -> 64.04 | Hurt |
| Replace natural-data pretraining with a short natural-training warm-up (ablation) | CURE on CIFAR-100, ResNet-18 · CIFAR-100 | NRR: 38.27 -> 36.04 | Hurt |
| Use a naturally pretrained model in the baseline instead of training without pretraining (ablation) | SAT and TRADES on CIFAR-10 and CIFAR-100 · CIFAR-10 and CIFAR-100 | accuracy | No clear effect |
| Set γ to 0.5 instead of 1.0 (ablation) | CURE on CIFAR-10, ResNet-18 · CIFAR-10 | robust accuracy: 54.92 -> 53.13 | Hurt |
| Set p to 10 instead of 30 (ablation) | CURE on CIFAR-10, ResNet-18 · CIFAR-10 | robust accuracy: 54.92 -> 55.65 | Helped |
| Set α to 0.2 instead of 0.1 (ablation) | CURE on CIFAR-10, ResNet-18 · CIFAR-10 | robust accuracy: 54.92 -> 54.17 | Hurt |
| Use CURE instead of the strongest listed baseline, LAS-AT | WideResNet-34-10 · CIFAR-10 | NRR: 67.19 -> 67.60 | Helped |
| Use CURE instead of the strongest listed baseline, FAT | ResNet-18 · CIFAR-10 | NRR: 63.41 -> 65.04 | Helped |
| Use CURE instead of the strongest listed baseline, ST-AT | ResNet-18 · CIFAR-100 | NRR: 36.65 -> 38.27 | Helped |
Surprising findings
- Updating only early layers reduces overfitting but lowers accuracy, while freezing early layers and updating later ones gives a milder accuracy decline.
- Baselines trained from a pretrained natural model do not improve much, suggesting CURE's gains come from how the pretrained knowledge is retained.
Limitations
- Training requires an additional pass to compute RGP scores, adding computation cost.
- Main hyperparameters p and alpha trade off natural versus robust accuracy and need tuning per setting.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN
Benchmarks: CIFAR-10 CIFAR-100 SVHN CIFAR-10-C
Metrics: natural accuracy PGD20 accuracy PGD100 accuracy AutoAttack accuracy C&W accuracy Natural-Robustness Ratio (NRR) robust overfitting gap (Delta)
Baselines: AT TRADES MART FAT ST-AT ACT ARD IAD LAS-AT AWP KD IDBH
Builds on: TRADES PGD adversarial training MART AutoAttack
Closest papers by meaning
- The Effectiveness of Random Forgetting for Robust Generalization (ICLR 2024 · similarity 0.89)
- Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation (NeurIPS 2024 · similarity 0.87)
- Adversarial Training Should Be Cast as a Non-Zero-Sum Game (ICLR 2024 · similarity 0.87)
- Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization (ICLR 2024 · similarity 0.87)
- Enhancing Adversarial Transferability with Checkpoints of a Single Model's Training (CVPR 2025 · similarity 0.87)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.