Research map / Learning theory and algorithms / Optimal transport
Stability Evaluation through Distributional Perturbation Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an optimal-transport-based stability criterion, defined as the minimal distributional perturbation (sample and density in a lifted space) needed to push a model's risk past a threshold, with convex dual reformulations and feature-level stability analysis.
Problem
Evaluating how stable a trained learning model is under out-of-sample distribution shifts, covering both data corruptions and sub-population shifts in a unified way.
Approach
Lifts the sample space to a joint (sample, density) space and measures the minimal OT discrepancy with moment constraints needed to raise the risk above a threshold; a strong duality theorem yields tractable convex/one-dimensional dual forms for piecewise-linear and general losses, with a tunable trade-off between corruption cost (theta1) and reweighting cost (theta2).
Key results
- Targeted adversarial training on the identified sensitive racial feature markedly outperforms ERM, LR, and AT baselines on worst racial group accuracy in income prediction (MLP worst-group accuracy drops from 82% to 72%).
- Per-age-group LR models substantially improve macro F1 for younger and older COVID-19 patients, after the criterion flagged over-reliance on age.
- Adversarial training yields higher stability than ERM for data corruptions, while Tilted ERM is more stable under sub-population shifts (income prediction, MLP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use targeted adversarial training on the identified sensitive racial feature, “American Indian” | Income prediction models · ACS Income, worst-performing racial group | accuracy | Helped |
| Use targeted adversarial training | Public coverage prediction models · ACS PubCov, worst-performing racial group | accuracy | Helped |
| Train distinct LR models for each age group | LR mortality prediction model · COVID-19 mortality prediction, age groups | macro F1 score | Helped |
| Use adversarial training rather than ERM | MLP model · ACS Income, distribution shifts | stability | Helped |
| Use adversarial training rather than Tilted ERM | MLP model · ACS Income, data corruptions | stability | Helped |
| Use Tilted ERM rather than adversarial training | MLP model · ACS Income, sub-population shifts | stability | Helped |
Surprising findings
- Robust learning methods tailored to one type of shift (AT for corruptions, Tilted ERM for sub-population shifts) do not generalize well to the other type of shift.
Limitations
- Nonlinear losses require nested inner adversarial maximization, which is computationally demanding.
- Experiments are limited to tabular datasets and small MLP/LR models; modern architectures and LLMs are not evaluated.
Experimental setup
Datasets: ACS Income ACS Public Coverage COVID-19 (SIVEP-Gripe, Brazil)
Benchmarks: ACS Income ACS Public Coverage COVID-19 mortality
Metrics: stability measure R(beta, r) excess risk decomposition worst racial group accuracy accuracy macro F1
Baselines: ERM AT Tilted ERM targeted AT
Builds on: Blanchet et al. 2023 OT discrepancy with moment constraints Namkoong et al. 2022 Gupta & Rothenhaeusler 2023 Sinha et al. 2018 adversarial training Li et al. 2023 Tilted ERM
Closest papers by meaning
- The s-value: evaluating stability with respect to distributional shifts (NeurIPS 2023 · similarity 0.88)
- The Bayesian Stability Zoo (NeurIPS 2023 · similarity 0.86)
- The Memory-Perturbation Equation: Understanding Model's Sensitivity to Data (NeurIPS 2023 · similarity 0.85)
- Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport (NeurIPS 2024 · similarity 0.84)
- Algorithmic Stability Unleashed: Generalization Bounds with Unbounded Losses (ICML 2024 · similarity 0.84)
- Achievable distributional robustness when the robust risk is only partially identified (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.