Research map / Adversarial robustness and security / Certified and empirical robustness
Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARWIN, an adversarially robust knowledge distillation method that aligns student and teacher predictions on intermediate adversarial samples from both untargeted and targeted adversarial paths, with adaptive weighting and separate BN layers.
Problem
Adversarially robust knowledge distillation compresses robust large teachers into lightweight students, but existing methods ignore intermediate adversarial samples along the multi-step adversarial path and only use untargeted adversaries, missing inter-class decision boundary structure.
Approach
Augment distillation with intermediate samples from untargeted (gradient ascent away from the true class) and targeted (gradient descent toward a chosen class) adversarial paths, weight each intermediate sample adaptively by index and student-teacher discrepancy, and add a triplet-style dual-branch loss that attracts same-branch predictions and repels cross-branch ones, with separate BN for target
Key results
- On CIFAR-10 with WRN-34 teacher and ResNet-18 student, DARWIN reaches 84.48% natural and 52.24% AutoAttack accuracy, versus 82.95%/49.46% for ARD.
- On ImageNet-100 with ResNet-18 student, DARWIN achieves 68.76% natural and 33.58% AutoAttack accuracy, above RSLAD (66.60%/32.18%).
- In black-box model extraction (CIFAR-10 to CIFAR-100 pretraining), DARWIN-LF reaches 46.29% natural and 21.90% robust accuracy versus 44.41%/18.51% for RSLAD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DARWIN instead of CRDND | WRN-34→ResNet-18 · CIFAR-10 | Natural accuracy: 83.92 -> 84.48 | Helped |
| Use DARWIN instead of RSLAD | WRN-34→ResNet-18 · CIFAR-10 | PGD-20 accuracy: 53.91 -> 55.07 | Helped |
| Use DARWIN instead of RSLAD | WRN-34→ResNet-18 · CIFAR-100 | AA accuracy: 26.62 -> 27.26 | Helped |
| Use DARWIN instead of RSLAD | ResNet-34→ResNet-18 · ImageNet-100 | Natural accuracy: 66.60 -> 68.76 | Helped |
| Add IAKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-10 | AA accuracy: 50.66 -> 51.70 | Helped |
| Add IAKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-100 | AA accuracy: 25.32 -> 26.95 | Helped |
| Add DBKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-10 | Natural accuracy: 83.09 -> 84.68 | Helped |
| Add DBKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-100 | PGD-20 accuracy: 30.59 -> 31.53 | Helped |
| Add both IAKD and DBKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-10 | PGD-20 accuracy: 53.95 -> 55.07 | Helped |
| Add both IAKD and DBKD to ARKD (ablation) | WRN-34→ResNet-18 · CIFAR-100 | AA accuracy: 25.32 -> 27.26 | Helped |
| Remove DBKD from ARKD + IAKD (ablation) | WRN-34→ResNet-18 · CIFAR-10 | Natural accuracy: 84.48 -> 82.74 | Hurt |
| Remove IAKD from ARKD + DBKD (ablation) | WRN-34→ResNet-18 · CIFAR-10 | PGD-20 accuracy: 55.07 -> 54.36 | Hurt |
Surprising findings
- For ResNet-18 students in self-distillation, the boundary risk gain falls below natural risk, leading to a slight natural accuracy drop relative to the teacher, as the theory predicts.
Limitations
- Requires ground-truth labels for targeted adversary generation in the main version (DARWIN-LF removes this with teacher predictions).
- Additional computation from multi-step intermediate adversaries and dual branches; costs not analyzed in the text shown.
- Natural-performance drop for some student-teacher pairs when boundary risk gain is below natural risk.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 TinyImageNet DDPM-generated auxiliary images (1M)
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 TinyImageNet AutoAttack
Metrics: natural accuracy PGD-20 robust accuracy CW robust accuracy AutoAttack robust accuracy
Baselines: ARD IAD RSLAD CRDND GACD TRADES AT-PRM
Builds on: TRADES ARD (adversarially robust distillation) RSLAD PGD adversarial training Zhang et al. robust risk decomposition
Closest papers by meaning
- Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.90)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.90)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.90)
- Indirect Gradient Matching for Adversarial Robust Distillation (ICLR 2025 · similarity 0.89)
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.89)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.