Research map / Adversarial robustness and security / Certified and empirical robustness
Provable Robust Overfitting Mitigation in Wasserstein Distributionally Robust Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Statistically Robust WDRO (SR-WDRO), an ambiguity set combining a Wasserstein adversarial budget with a KL statistical budget, to mitigate robust overfitting in adversarial training, with generalization bounds and equilibrium existence results.
Problem
Robust overfitting in adversarial training and Wasserstein distributionally robust optimization (WDRO), where test adversarial robustness degrades after the first learning-rate decay because WDRO ignores statistical error from finite samples.
Approach
Define the ambiguity set as distributions within KL divergence gamma of a Wasserstein-epsilon ball around the empirical distribution, so the training loss covers both adversarial noise and statistical error; the resulting loss is computed approximately by re-weighting UDR adversarial losses under a KL constraint.
Key results
- On CIFAR-10 with ResNet-18 (l_inf 8/255, PGD-10), SR-WDRO reaches 48.58% final robust accuracy vs 47.32% for HR and 46.60% for UDR-AT, with the smallest best-to-final gap (3.36 points).
- On CIFAR-100, SR-WDRO reaches 23.09% final robust accuracy vs 22.07% for UDR-AT and 21.15% for HR, with the smallest gap (5.86 points).
- On CIFAR-10 with WideResNet28-10, SR-WDRO reaches 51.26% final robust accuracy with a gap of 2.61 points, versus 49.01% for UDR-AT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the KL-based statistical robustness budget to WDRO (SR-WDRO) | UDR-style robust training on ResNet-18 · CIFAR-10, PGD-10, ε=8/255 | final robust test accuracy (%): 47.32 ± 0.59 -> 48.58 ± 0.21 | Helped |
| Add the KL-based statistical robustness budget to WDRO (SR-WDRO) | UDR-style robust training on ResNet-18 · CIFAR-100, PGD-10, ε=8/255 | final robust test accuracy (%): 22.07 ± 0.12 -> 23.09 ± 0.20 | Helped |
| Add the KL-based statistical robustness budget to WDRO (SR-WDRO) | UDR-style robust training on WideResNet28-10 · CIFAR-10, PGD-10, ε=8/255 | final robust test accuracy (%): 49.01 ± 0.13 -> 51.26 ± 0.42 | Helped |
| Add the KL-based statistical robustness budget to WDRO (SR-WDRO) | robust training methods on ResNet-18 · CIFAR-10, PGD-200, ε=8/255 | robust accuracy (%): 44.59 ± 0.27 -> 46.79 ± 0.11 | Helped |
| Add the KL-based statistical robustness budget to WDRO (SR-WDRO) | robust training methods on ResNet-18 · CIFAR-10, Auto-Attack, ε=8/255 | robust accuracy (%): 42.81 ± 0.24 -> 44.06 ± 0.32 | Helped |
| Increase statistical budget γ (ablation) | SR-WDRO · CIFAR-10, training over epochs | robust overfitting | Helped |
| Increase statistical budget from γ=0 to γ=0.2 (ablation) | SR-WDRO · CIFAR-10 | natural test accuracy (%): 84.8% -> 80.4% | Hurt |
| Use γ=0.2 rather than γ=0 (ablation) | SR-WDRO · CIFAR-10 | overall performance trade-off: 84.8% -> 80.4% | Mixed |
Surprising findings
- Increasing the statistical budget gamma reduces overfitting but lowers natural accuracy (84.8% at gamma=0 to 80.4% at gamma=0.2).
- On CIFAR-10 with ResNet-18, SR-WDRO's natural accuracy (83.34%) is lower than PGD-AT (84.80%).
Limitations
- Natural accuracy drops relative to baselines, and there is slight computational overhead.
- Exact statistically robust loss is intractable, so an approximate finite reformulation is used.
- Focus is restricted to supervised classification; unsupervised and regression tasks are not studied.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: robust test accuracy natural accuracy robust overfitting gap
Baselines: PGD-AT UDR-AT HR training TRADES
Builds on: Wasserstein distributionally robust optimization UDR adversarial training PGD adversarial training Auto-Attack
Closest papers by meaning
- Principled learning method for Wasserstein distributionally robust optimization with local perturbations (ICML 2020 · similarity 0.91)
- Outlier-Robust Wasserstein DRO (NeurIPS 2023 · similarity 0.91)
- A Unified Wasserstein Distributional Robustness Framework for Adversarial Training (ICLR 2022 · similarity 0.90)
- Universal generalization guarantees for Wasserstein distributionally robust models (ICLR 2025 · similarity 0.89)
- Knowledge-Guided Wasserstein Distributionally Robust Optimization (ICML 2025 · similarity 0.89)
- Adversarially Robust Hypothesis Transfer Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.