Research map / Adversarial robustness and security / Certified and empirical robustness

Indirect Gradient Matching for Adversarial Robust Distillation

ICLR 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IGDM, a plug-in module that indirectly matches a student's input gradient to a teacher's via logit differences under small perturbations, improving existing adversarial distillation methods on CIFAR-10/100, SVHN, and Tiny-ImageNet.

Problem

Small models trail large models in adversarial robustness; existing adversarial distillation (AD) methods match only teacher logits, leaving the teacher's input-gradient knowledge untransferred.

Approach

Exploiting local linearity of adversarially trained models, the student's and teacher's output differences f(x+δ)−f(x−δ) are matched, which implicitly aligns their input gradients without computing gradients directly; this IGDM loss is added to any AD method's outer loss.

Key results

• IGDM on AdaAD raises AutoAttack accuracy on CIFAR-100 with ResNet-18 and BDM-AT teacher from 28.06% to 30.32%.
• IGDM on MobileNetV2 with BDM-AT teacher (CIFAR-100) raises RSLAD AutoAttack from 26.18% to 29.32%.
• IGDM improves AA for most AD methods across CIFAR-10/100, SVHN and Tiny-ImageNet while keeping clean accuracy roughly unchanged, and reduces gradient distance between teacher and student.

Measured effects reported in the paper

Surprising findings

• Directly matching input gradients (Direct-ARD) fails to meaningfully improve robustness over ARD, while the indirect logit-difference formulation works well.
• Replacing AdaAD's inner maximization with a student-only PGD attack reduces robustness, yet IGDM without the teacher in inner maximization still outperforms the original AdaAD.

Limitations

• Gains depend on hyperparameter alpha, which must be tuned by grid search; large alpha can reduce clean accuracy or robustness.
• Gradient matching relies on the local linearity assumption, validated empirically only for small perturbations (8/255 uniform noise).

Experimental setup

Datasets: CIFAR-10 CIFAR-100 SVHN Tiny-ImageNet

Benchmarks: CIFAR-10 CIFAR-100 SVHN Tiny-ImageNet OODRobustBench

Metrics: clean accuracy FGSM accuracy PGD-20 accuracy C&W accuracy AutoAttack accuracy gradient distance gradient cosine similarity

Baselines: PGD-AT TRADES ARD IAD AKD RSLAD AdaAD AdaIAD

Builds on: adversarial robustness distillation (ARD) TRADES RSLAD AdaAD PGD adversarial training

Closest papers by meaning

• Annealing Self-Distillation Rectification Improves Adversarial Training (ICLR 2024 · similarity 0.89)
• Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.89)
• Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.89)
• Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.89)
• Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.87)
• Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.87)