Research map / Adversarial robustness and security / Certified and empirical robustness

Towards Adversarially Robust Dataset Distillation by Curvature Regularization

AAAI 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Robust overfitting and regularization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GUARD, a curvature-regularized dataset distillation method (added to SRe2L, DC, CDA) that yields distilled datasets whose trained models are more adversarially robust at lower cost than adversarial training.

Problem

Standard dataset distillation optimizes only clean test accuracy; naively embedding adversarial training into distillation hurts clean accuracy and gives inconsistent robustness gains, and robust distillation as a tri-level problem is computationally expensive.

Approach

A theoretical bound shows the adversarial loss of models trained on distilled data is governed by the loss curvature (largest Hessian eigenvalue) on real data; GUARD approximates this curvature with a finite-difference gradient-based regularizer along the normalized gradient direction, adding only one extra forward pass per iteration.

Key results

• On ImageNette at 10 IPC under PGD100, GUARD reaches 23.87% vs 4.76% for SRe2L and 39.85% for MTT (MTT is higher), while GUARD's clean accuracy is 57.93% vs SRe2L's 42.42%.
• Per-iteration time for GUARD is 0.007 s versus 2.198 s for adversarial training (batch 32, A100), about 300x faster.
• Adding GUARD to CIFAR10 SRe2L at 1 IPC raises clean accuracy from 17.13% to 22.88% and PGD100 accuracy from 13.56% to 19.21%.

Measured effects reported in the paper

Surprising findings

• GUARD improves clean accuracy as well as robustness, which the authors attribute to regularization aiding generalization.
• On ImageNette at 100 IPC, SRe2L's clean accuracy (85.83%) exceeds GUARD's (83.39%), and on ImageNet-1K at 10 and 50 IPC SRe2L's clean and Square accuracy is sometimes higher than GUARD's.

Limitations

• Theoretical bound relies on strong assumptions (convexity of the loss, linear feature extractor, Lipschitz adversarial loss) acknowledged by the authors.
• Main experiments integrate GUARD only into SRe2L; MTT and TESLA use ConvNet, and 100 IPC on ImageNet was not run for all methods due to compute limits.
• Robustness evaluated only at fixed epsilon=1/255 attacks; MTT outperforms GUARD on some clean and PGD metrics.

Experimental setup

Datasets: ImageNette Tiny ImageNet ImageNet-1K CIFAR10

Benchmarks: ImageNette Tiny ImageNet ImageNet-1K CIFAR10

Metrics: top-1 accuracy (clean) robust accuracy under attack

Baselines: SRe2L MTT TESLA SRe2L+adversarial training DC CDA gradient regularization

Builds on: SRe2L MTT DC CURE

Closest papers by meaning

• ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck (ICML 2025 · similarity 0.90)
• Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.90)
• Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.89)
• Indirect Gradient Matching for Adversarial Robust Distillation (ICLR 2025 · similarity 0.89)
• Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.89)
• Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation (NeurIPS 2024 · similarity 0.88)