Research map / Adversarial robustness and security / Black-box and transfer attacks
ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROME, an information-bottleneck-based dataset distillation method that adds performance- and robustness-aligned loss terms with adversarially perturbed source data, and a refined robustness metric I-RR, improving adversarial robustness of distilled CIFAR-10/100 datasets without adversarial retraining.
Problem
Distilled datasets make models trained on them vulnerable to adversarial attacks, and existing robustness remedies (adversarial training) are costly and trade off accuracy against robustness.
Approach
Frames robust dataset distillation as an information bottleneck (conditional entropy bottleneck) problem, optimizing synthetic images with a cross-entropy term from a robust pretrained model plus a feature-distribution alignment term against adversarially perturbed real images, so no adversarial retraining is needed.
Key results
- On CIFAR-10 IPC-50 under targeted white-box attacks, ROME reaches 81.36% RR and 97.44% I-RR, improving I-RR by 8.93 points over the best baseline DC.
- Under black-box transfer-based targeted attacks on CIFAR-10 IPC-50, ROME achieves 99.90% I-RR, 5.81 points above the second-best method.
- ROME reaches I-CREI 63.32% in the base setting without adversarial retraining, versus 58.21% for DC with adversarial fine-tuning reaching 63.43%, while training in 0.418 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ROME instead of the strongest listed baseline | CIFAR-10 targeted white-box evaluation · CIFAR-10 targeted white-box | I-RR: 88.51% -> 97.44% | Helped |
| use ROME instead of the strongest listed baseline | CIFAR-10 untargeted white-box evaluation · CIFAR-10 untargeted white-box | I-RR: 56.19% -> 67.01% | Helped |
| use ROME instead of the strongest listed baseline | CIFAR-100 targeted white-box evaluation · CIFAR-100 targeted white-box | I-RR: 77.14% -> 100.65% | Helped |
| use ROME instead of the strongest listed baseline | CIFAR-100 untargeted white-box evaluation · CIFAR-100 untargeted white-box | I-RR: 33.29% -> 46.24% | Helped |
| use ROME instead of the strongest listed baseline | CIFAR-10 black-box targeted transfer attacks · CIFAR-10 targeted transfer black-box | I-RR: 94.09% -> 99.90% | Helped |
| use ROME instead of the strongest listed baseline | CIFAR-10 black-box untargeted query attacks · CIFAR-10 untargeted query black-box | I-RR: 63.23% -> 78.46% | Helped |
| add a robust pretrained model (ablation) | ROME · CIFAR-10 targeted and untargeted white-box attacks | I-RR: 81.86% -> 84.50%; 32.45% -> 34.89% | Helped |
| add adversarial perturbation (ablation) | ROME · CIFAR-10 targeted and untargeted white-box attacks | I-RR: 81.86% -> 94.66%; 32.45% -> 47.64% | Helped |
| add robust pretrained model and adversarial perturbation (ablation) | ROME · CIFAR-10 targeted and untargeted white-box attacks | I-RR: 81.86% -> 97.73%; 32.45% -> 51.73% | Helped |
| increase α from 0.0 to 0.2 (ablation) | ROME · CIFAR-10 IPC-50 | accuracy | Mixed |
Surprising findings
- On CIFAR-100 under targeted attacks, I-RR exceeds 100% (up to 103.09%), which the authors call an 'Over-Robustness Phenomenon'.
- Targeted-attack robustness is reported with I-RR values above 100%, meaning models trained on ROME can score higher under attack than on clean data.
Limitations
- Scalability limited by large search space for complex datasets like ImageNet.
- Applicability to large-scale language and vision-language models is restricted.
- Requires a robust pretrained model and adversarial perturbations to build the robust prior, with robustness trading off against clean accuracy as alpha increases.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: I-RR I-CREI RR CREI accuracy under PGD
Baselines: DC DSA MTT DM IDM BACON
Builds on: Dataset Condensation (DC) Distribution Matching (DM) Conditional Entropy Bottleneck BEARD Information Bottleneck Distillation (IBD)
Closest papers by meaning
- Improving Adversarial Robustness via Information Bottleneck Distillation (NeurIPS 2023 · similarity 0.92)
- Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust? (ICML 2026 · similarity 0.91)
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.90)
- Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.90)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.89)
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.