Research map / Adversarial robustness and security / Certified and empirical robustness

Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement

CVPR 2024 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training refinements · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Guided Label Refinement (SGLR), which softens hard labels using the model's own EMA-smoothed predictions to mitigate robust overfitting in adversarial training without external teachers.

Problem

Adversarial training suffers from robust overfitting, a large gap between best and final robust test accuracy, which the paper attributes to memorization of noisy over-confident hard labels.

Approach

Replace one-hot labels with soft labels formed by interpolating hard labels and an exponential-moving-average of the model's own predictions (self-distillation without an external teacher), which the paper argues reduces information in weights and memorization of noisy labels.

Key results

• On CIFAR-10 with ResNet-18, PGD-AT+SGLR reaches 56.4% best PGD-20 robust accuracy and reduces the best-final PGD-20 gap from 9.3% to 0.5%.
• Under AutoAttack on CIFAR-10, PGD-AT+SGLR achieves 51.2% best robust accuracy versus 47.7% for PGD-AT, with gap 1.0 vs 7.5.
• On CIFAR-100, TRADES+SGLR improves AutoAttack robust accuracy from 24.6% to 27.1% (best) and narrows the gap from 1.2 to 0.7.

Measured effects reported in the paper

Surprising findings

• Plain label smoothing in adversarial training fails to mitigate robust overfitting (PGD-AT+LS gap 4.8% PGD-20) while SGLR does.
• Increasing the softmax temperature gives only slight standard accuracy gains and degrades robust accuracy.

Limitations

• Model remains poorly calibrated, since calibration is measured over the whole dataset rather than per sample.
• Theoretical noise-tolerance results rely on symmetric or asymmetric noise assumptions and an approximation (≲) in the information-theoretic analysis.

Experimental setup

Datasets: CIFAR-10 CIFAR-100

Benchmarks: CIFAR-10 CIFAR-100

Metrics: natural accuracy PGD-20 robust accuracy AutoAttack robust accuracy best-final generalization gap

Baselines: PGD-AT TRADES PGD-AT+LS PGD-AT+TE AWP KD-AT KD-SWA ARD RSLAD

Builds on: PGD-AT TRADES label smoothing symmetric cross entropy adversarial weight perturbation

Closest papers by meaning

• Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting (NeurIPS 2022 · similarity 0.93)
• Annealing Self-Distillation Rectification Improves Adversarial Training (ICLR 2024 · similarity 0.90)
• Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks (NeurIPS 2022 · similarity 0.90)
• Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation (NeurIPS 2024 · similarity 0.89)
• Toward Improving Robustness and Accuracy in Unsupervised Domain Adaptation (AAAI 2025 · similarity 0.88)
• Training Deep Neural Networks with Virtual Smoothing Classes (AAAI 2025 · similarity 0.88)