Research map / Adversarial robustness and security / Certified and empirical robustness
Annealing Self-Distillation Rectification Improves Adversarial Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADR, which builds rectified soft labels from a self-distillation EMA teacher with annealed temperature and interpolation toward one-hot labels, to mitigate robust overfitting in adversarial training.
Problem
Adversarial training with one-hot labels is overconfident and causes robust overfitting, and existing label-softening fixes either assume uniform noise or need a pre-trained robust teacher.
Approach
Generate noise-aware soft labels from the EMA (mean-teacher) of the model being trained, softened with an annealed temperature (high to low) and interpolated with one-hot labels using an increasing ratio, with per-sample adjustment so the true class stays dominant; plug the labels into PGD-AT, TRADES, WA and AWP without any pre-trained model.
Key results
- On CIFAR-10 with ResNet-18, AT+ADR raises AutoAttack robust accuracy from 48.81% to 50.38% and shrinks the robust overfitting gap from 5.62% to 3.20%.
- AT+WA+AWP+ADR reaches 28.50% AutoAttack robust accuracy on CIFAR-100 with ResNet-18 (no extra data), above the compared methods in the table.
- On CIFAR-100 with WRN-34-10, AT+ADR with 1M DDPM data reaches 32.19% robust accuracy versus 31.15% for the best prior method with extra data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace hard labels with ADR rectified labels in AT | AT with ResNet-18 · CIFAR-10 | AutoAttack accuracy (Best, %): 48.81 -> 50.38 | Helped |
| Replace hard labels with ADR rectified labels in TRADES | TRADES with ResNet-18 · CIFAR-10 | AutoAttack accuracy (Best, %): 50.1 -> 51.02 | Helped |
| Replace hard labels with ADR rectified labels in AT | AT with ResNet-18 · CIFAR-100 | AutoAttack accuracy (Best, %): 24.95 -> 26.87 | Helped |
| Replace hard labels with ADR rectified labels in TRADES | TRADES with ResNet-18 · CIFAR-100 | AutoAttack accuracy (Best, %): 24.71 -> 26.42 | Helped |
| Replace hard labels with ADR rectified labels in AT | AT with ResNet-18 · TinyImageNet-200 | AutoAttack accuracy (Best, %): 18.06 -> 19.46 | Helped |
| Replace hard labels with ADR rectified labels in TRADES | TRADES with ResNet-18 · TinyImageNet-200 | AutoAttack accuracy (Best, %): 17.35 -> 19.17 | Helped |
| Add ADR to AT with WA and AWP | AT + WA + AWP, ResNet-18 · CIFAR-10 | AutoAttack accuracy: + 2.37 | Helped |
| Add ADR to AT with WA and AWP | AT + WA + AWP, ResNet-18 · CIFAR-100 | AutoAttack accuracy: + 3.55 | Helped |
| Add ADR to AT with WA and AWP | AT + WA + AWP, WRN-34-10 · TinyImageNet-200 | AutoAttack accuracy: + 2.57 | Helped |
| Anneal temperature and interpolation factor instead of holding them constant (ablation) | ADR · Adversarial training benchmarks | robust accuracy | Helped |
| Use a fixed temperature and interpolation factor rather than annealing (ablation) | ADR · Adversarial training benchmarks | robust accuracy | Hurt |
| Add ADR to TRADES with WA and AWP (ablation) | TRADES + WA + AWP, ResNet-18 · CIFAR-10 | AutoAttack robust accuracy: 51.25 -> 50.59 | Hurt |
Surprising findings
- ADR's gains saturate or slightly degrade when combined with WA and AWP on CIFAR-10, attributed to TRADES already using soft targets.
- Initializing ADR from an AT pre-trained checkpoint did not increase maximum robustness, though it raised standard accuracy by 1.25%.
Limitations
- Optimal temperature and interpolation schedules are dataset-dependent and must be hand-tuned.
- Gains saturate when combined with WA and AWP on CIFAR-10 for TRADES-based training.
- WRN-34-10 reused ResNet-18 hyperparameters, which may be suboptimal and caused some standard accuracy drops.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet-200 DDPM synthetic dataset (1M samples)
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet-200 AutoAttack
Metrics: robust accuracy (AutoAttack) standard accuracy robust overfitting gap (best vs final)
Baselines: PGD-AT TRADES AWP ARD RSLAD IAD MT WA
Builds on: PGD-AT TRADES Weight Average AWP
Closest papers by meaning
- Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement (CVPR 2024 · similarity 0.90)
- Indirect Gradient Matching for Adversarial Robust Distillation (ICLR 2025 · similarity 0.89)
- Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation (NeurIPS 2024 · similarity 0.88)
- 3SAT: A Simple Self-Supervised Adversarial Training Framework (AAAI 2025 · similarity 0.88)
- Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.87)
- Self-Supervised Adversarial Training via Diverse Augmented Queries and Self-Supervised Double Perturbation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.