Research map / Adversarial robustness and security / Certified and empirical robustness

Long-tailed Adversarial Training with Self-Distillation

ICLR 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training refinements

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a two-step self-distillation method for adversarial training on long-tailed data: train a PGD-AT balanced self-teacher on a class-balanced resampled subset, then distill its tail-class robustness into a balanced-softmax adversarial student.

Problem

Adversarial training on long-tailed distributions yields poor robustness and accuracy on tail classes, which existing long-tailed adversarial training methods (combining balanced softmax with AT) do not adequately address.

Approach

Build a class-balanced sub-dataset by resampling the long-tailed data, adversarially train a self-teacher on it, and distill this teacher into the main long-tailed adversarial student (with balanced softmax loss) via adversarial knowledge distillation to improve tail-class robustness.

Key results

• On CIFAR-10-LT (ResNet-18, best checkpoint) the method reaches 70.81 clean, 38.85 PGD, 34.32 AA, and 36.9 tail PGD-20 accuracy, versus 35.98 PGD for REAT (best baseline PGD).
• On CIFAR-100-LT (ResNet-18, last checkpoint) the method reports 47.22 clean, 21.82 PGD, 18.53 AA, and 17.6 tail T-PGD, improving tail T-PGD over AT-BSL's 9.1 in the last checkpoint setting.
• Abstract claims tail-class PGD accuracy gains of 20.3, 7.1, and 3.8 points on CIFAR-10, CIFAR-100, and Tiny-ImageNet-LT.

Measured effects reported in the paper

Surprising findings

• Naive adversarial training (PGD-AT, TRADES, MART, AWP) on long-tailed data gives reasonable overall robustness but very poor tail-class accuracy, and the paper argues tail robustness under AT can fall below natural training.
• The best results were obtained using a PGD-AT teacher trained on the balanced subset for only 30 epochs, even though that teacher had the lowest overall accuracy.

Limitations

• Evaluation focuses on data quantity; class-level fairness and intrinsic class difficulty are not addressed.
• Teacher is trained with a simple PGD-AT; developing a more effective teacher is left for future work.
• Tail-class evaluation groups are defined by fixed class counts (last class for CIFAR-10-LT, tail 10/20 classes otherwise), and gains on the best checkpoint depend on checkpoint selection by PGD accuracy.

Experimental setup

Datasets: CIFAR-10-LT CIFAR-100-LT Tiny-ImageNet-LT

Benchmarks: CIFAR-10-LT CIFAR-100-LT Tiny-ImageNet-LT

Metrics: clean accuracy PGD-20 accuracy AutoAttack accuracy tail-class clean accuracy (T-Clean) tail-class PGD-20 accuracy (T-PGD)

Baselines: PGD-AT TRADES MART AWP RoBal REAT AT-BSL

Builds on: PGD adversarial training TRADES balanced softmax RoBal AT-BSL

Closest papers by meaning

• TAET: Two-Stage Adversarial Equalization Training on Long-Tailed Distributions (CVPR 2025 · similarity 0.92)
• Revisiting Adversarial Training Under Long-Tailed Distributions (CVPR 2024 · similarity 0.92)
• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.88)
• Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions (CVPR 2024 · similarity 0.88)
• Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.88)
• Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.88)