Research map / Adversarial robustness and security / Certified and empirical robustness
Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Data-Free Adversarial Robustness Distillation (DFARD), training small robust students without any training data via an Interactive Temperature Adjustment strategy and an Adaptive Generator Balance module.
Problem
Adversarial robustness distillation normally requires original or public training data, which is often private or unavailable in high-security settings; the paper asks how to distill robustness from a teacher without data.
Approach
Adapts the distillation temperature per epoch from the gap between teacher and student confidence (ITA) and sets the generator loss weight adaptively from the teacher's mean confidence (AGB), so no hand-tuned temperature or loss weights are needed in a data-free adversarial distillation pipeline.
Key results
- On CIFAR-10 with RN-18 student, the full method reaches 35.31% average robust accuracy versus 29.06% for DFAD and 28.05% for Fast among baselines (per the table; best baseline improvement reported as 1.98%).
- On CIFAR-100 with RN-18, the method achieves 22.08% average robust accuracy, above the best baseline Fast/DAFL at about 19.50%.
- Adding ITA to DAFL, DFQ and ZSKT improves their AutoAttack accuracy by about 1.9 to 2.2 points on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full DFARD with ITA and AGB (ablation) | RN-18 student · CIFAR-10 | average adversarial accuracy: 27.53 -> 35.31 | Helped |
| Add ITA in the generation stage (ablation) | Vanilla DFARD, RN-18 student · CIFAR-10 | average adversarial accuracy: 27.53 -> 27.72 | Helped |
| Apply ITA in generation and distillation instead of generation only (ablation) | DFARD with ITA (G), RN-18 student · CIFAR-10 | average adversarial accuracy: 27.72 -> 29.82 | Helped |
| Add AGB to vanilla DFARD (ablation) | RN-18 student · CIFAR-10 | average adversarial accuracy: 27.53 -> 33.05 | Helped |
| Add ITA to DAFL in generation and distillation (ablation) | DAFL · CIFAR-10, RN-18 | AA accuracy: +2.11 | Helped |
| Add ITA to DFQ in generation and distillation (ablation) | DFQ · CIFAR-10, RN-18 | AA accuracy: +2.17 | Helped |
| Add ITA to ZSKT in generation and distillation (ablation) | ZSKT · CIFAR-10, RN-18 | AA accuracy: +1.90 | Helped |
| Use ITA instead of a fixed generator temperature (ablation) | RN-18 student · CIFAR-10 | AA accuracy: 18.39 -> 18.42 | Helped |
| Replace adaptive AGB with fixed λ 0.9 (ablation) | RN-18 student · CIFAR-10 | average adversarial accuracy: 9.20 vs 33.11 | Hurt |
| Use full DFARD instead of the strongest baseline | RN-18 student · CIFAR-10 | average adversarial accuracy | Helped |
| Use full DFARD instead of the strongest baseline | RN-18 student · CIFAR-100 | average adversarial accuracy | Helped |
| Use full DFARD instead of the strongest baseline | MN-V2 student · CIFAR-10 | average adversarial accuracy | Helped |
Surprising findings
- Students trained with a larger teacher (e.g., WRN-70-16) saturate for baselines, while the proposed method is less sensitive to the teacher capacity gap.
Limitations
- Relies on a fixed PGD-based adversarial generation and a pretrained robust teacher; the paper evaluates only CIFAR-10 and CIFAR-100 at 32x32 resolution.
- Some baseline hyperparameters are copied from original papers and the best temperature in the vanilla setting is chosen by sweeping, so the fairness of comparisons depends on these settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: clean accuracy robust accuracy under FGSM, PGD-S, PGD-T, CW-inf, AutoAttack average robust accuracy GPU synthesis time
Baselines: Dream DeepInv DAFL DFAD ZSKT DFQ CMI Fast
Builds on: Adversarial Robustness Distillation (ARD) DAFL DFAD ZSKT DFQ CMI Fast Dream DeepInv
Closest papers by meaning
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.90)
- Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.90)
- ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck (ICML 2025 · similarity 0.90)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.90)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.89)
- Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.