Research map / Adversarial robustness and security / Certified and empirical robustness
Two is Better than One: Efficient Ensemble Defense for Robust and Compact Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Efficient Ensemble Defense (EED), which builds an adversarially robust compact ensemble from sub-models of a single base model pruned with different importance scores, with robust-diversity-based selection and dynamic inference ensembling.
Problem
Compressed (adversarially pruned) models are highly vulnerable to adversarial attacks, and traditional ensemble defenses improve robustness only at the cost of multiple large base models.
Approach
Generate diverse sub-models by pruning one base model with different importance scores and data subsets, select ensembles by a Robust Diversity score, train with ensemble, misclassification and compactness regularizers, and add sub-models dynamically at inference via a geometric stopping rule.
Key results
- At 80% sparsity on CIFAR-10 with ResNet-18, EED reaches 55.71% PGD and 48.13% AutoAttack accuracy with 1.86x speed-up, versus 52.56% PGD and 45.36% AA for HARP.
- On SVHN with ResNet-18 at 80% sparsity, EED achieves 55.74% PGD accuracy versus 54.16% for HARP, with 1.85x speed-up.
- Dynamic inference ensemble at 50% sparsity improves PGD by 1.81% and AA by 1.59% while raising speed-up from 1.27x to 1.64x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EED instead of AT | ResNet18 · CIFAR-10, sr=80% | AA accuracy: 48.02 -> 48.13 | Helped |
| Use EED instead of AT | ResNet18 · SVHN, sr=80% | PGD accuracy: 56.27 -> 55.74 | No clear effect |
| Add misclassification regularization (ablation) | EED · CIFAR-10, ResNet18, sr=80% | AA accuracy: 36.37 -> 48.13 | Helped |
| Add compactness regularization (ablation) | EED · CIFAR-10, ResNet18, sr=80% | PGD accuracy: 39.10 -> 55.71 | Helped |
| Add misclassification and compactness regularization (ablation) | EED · CIFAR-10, ResNet18, sr=80% | AA accuracy: 28.94 -> 48.13 | Helped |
| Enable dynamic inference ensemble (ablation) | EED · CIFAR-10, ResNet18, sr=80% | PGD accuracy: 55.62 -> 55.71 | No clear effect |
| Enable dynamic inference ensemble (ablation) | EED · CIFAR-10, ResNet18, sr=80% | Speedup: 1.80x -> 1.86x | Helped |
| Enable dynamic inference ensemble (ablation) | EED · CIFAR-10, ResNet18, sr=50% | PGD accuracy: 54.28 -> 56.09 | Helped |
| Enable dynamic inference ensemble (ablation) | EED · CIFAR-10, ResNet18, sr=50% | AA accuracy: 50.76 -> 52.35 | Helped |
| Increase sparsity rate (ablation) | EED · CIFAR-10, ResNet18 | Robust accuracy | Hurt |
| Use EED instead of existing AP methods | ResNet18 · CIFAR-10, sr=80% | PGD accuracy | Helped |
Surprising findings
- Ablation shows that removing the compactness regularizer (gamma=0) substantially degrades attack accuracy, as ensemble size must shrink further to meet sparsity.
Limitations
- Robustness declines as sparsity rate increases because ensemble diversity drops at high compression.
- Evaluated only on CIFAR-10 and SVHN with two CNN architectures.
Experimental setup
Datasets: CIFAR-10 SVHN
Benchmarks: CIFAR-10 SVHN
Metrics: clean accuracy PGD accuracy AutoAttack accuracy C&W accuracy DeepFool accuracy inference speed-up
Baselines: AT R-ADMM HYDRA RST MAD Flying Bird HARP TwinRep
Builds on: Madry adversarial training HARP iGAT Inference-efficient ensemble (Wings of Certainty / WoC) DVERGE
Closest papers by meaning
- Adversarial Defence by Diversified Simultaneous Training of Deep Ensembles (AAAI 2021 · similarity 0.87)
- Improving Adversarial Robustness via Promoting Ensemble Diversity (ICML 2019 · similarity 0.86)
- MORA: Improving Ensemble Robustness Evaluation with Model Reweighing Attack (NeurIPS 2022 · similarity 0.86)
- Ensemble-in-One: Ensemble Learning within Random Gated Networks for Enhanced Adversarial Robustness (AAAI 2023 · similarity 0.86)
- Building Robust Ensembles via Margin Boosting (ICML 2022 · similarity 0.85)
- Adversarial Robustness via Deformable Convolution with Stochasticity (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.