Research map / Adversarial robustness and security / Certified and empirical robustness
Adversarial Distillation Based on Slack Matching and Attribution Region Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmaraAD, an adversarial distillation method that replaces KL matching with a differentiable Spearman-correlation "slack matching" and aligns CAM attribution regions between teacher and student, improving robustness of small models on CIFAR-10/100.
Problem
Adversarial distillation for small models is hurt by exact KL-based output matching when teacher and student predictions differ substantially, and by output-only matching that fails to transfer the teacher's behaviour; higher-robustness teachers thus do not yield more robust students.
Approach
Replace KL divergence with a soft-ranking Spearman correlation objective (slack matching) for prediction transfer, and add an MSE alignment between CAM attribution maps of student and teacher, with adversarial examples generated by maximizing KL plus attribution loss and a teacher-confidence/entropy-based sample weighting (SmaraAD++).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SmaraAD++ instead of AdaAD | ResNet-18 · CIFAR-10, WideResNet-34-10 teacher | AA accuracy: 51.41% -> 53.40% | Helped |
| Use SmaraAD++ instead of AdaAD | MobileNet-V2 · CIFAR-10, WideResNet-34-10 teacher | AA accuracy: 49.53% -> 52.54% | Helped |
| Use SmaraAD++ instead of RSLAD | ResNet-18 · CIFAR-100, WideResNet-34-10 teacher | AA accuracy: 27.56% -> 30.63% | Helped |
| Use SmaraAD++ instead of AdaAD | MobileNet-V2 · CIFAR-100, WideResNet-34-10 teacher | AA accuracy: 27.01% -> 29.65% | Helped |
| Add attribution region alignment to slack matching (ablation) | SmaraAD, ResNet-18 · CIFAR-10 | robust accuracy | Helped |
| Replace KL divergence with Spearman slack matching (ablation) | SmaraAD, ResNet-18 · CIFAR-10 | robust accuracy | Helped |
Limitations
- Evaluated only on CIFAR-10 and CIFAR-100 with two small student architectures.
- Attribution alignment relies on CAM, which requires a classification layer structure and is specific to image classification.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: clean accuracy robust accuracy
Closest papers by meaning
- Indirect Gradient Matching for Adversarial Robust Distillation (ICLR 2025 · similarity 0.91)
- Adversarial Moment-Matching Distillation of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.89)
- Annealing Self-Distillation Rectification Improves Adversarial Training (ICLR 2024 · similarity 0.88)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.88)
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.