Research map / Optimization / Nonconvex and smooth optimization
Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops SFK-DRO, a stochastic primal-Frank-Wolfe algorithm for non-convex constrained DRO under Cressie-Read divergences, with per-iteration cost independent of dataset size.
Problem
Constrained distributionally robust optimization with non-convex loss (e.g., neural networks) over Cressie-Read divergence uncertainty sets, at large dataset scale where per-iteration cost must not grow with sample size.
Approach
Reformulates the constrained DRO dual over a known distribution, approximates it with a smooth Lipschitz version by bounding the dual variables in a box with a small regularizer lambda_0, then runs mini-batch SGD on the model x and Frank-Wolfe on the dual variables z=(lambda, eta), with a subsampling bias bound for the min-of-expectation estimate.
Key results
- Theorem gives an epsilon-stationary point with complexity O(epsilon^{-3k_*-5}) for the stated method, per the abstract.
- On imbalanced CIFAR-10, SFK-DRO reaches 58.65% on class 4 versus 51.53% for ERM and 55.83% for PAN-DRO; overall accuracy is comparable.
- In the experiments, SFK-DRO's dual objective converges faster in CE loss than PGD optimizing the same objective, though the paper's own complexity for PGD matches SFK-DRO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use constrained SFK-DRO instead of ERM (EMR) | Alexnet · Imbalanced CIFAR-10 class 3 test | accuracy: 54.03 -> 54.95 | Helped |
| Use constrained SFK-DRO instead of ERM (EMR) | Alexnet · Imbalanced CIFAR-10 class 4 test | accuracy: 51.53 -> 58.65 | Helped |
| Use constrained SFK-DRO instead of ERM (EMR) | Alexnet · Imbalanced CIFAR-10 class 5 test | accuracy: 47.05 -> 49.36 | Helped |
| Use constrained SFK-DRO instead of penalized PAN-DRO | Alexnet · Imbalanced CIFAR-10 class 3 test | accuracy: 52.69 -> 54.95 | Helped |
| Use constrained SFK-DRO instead of penalized PAN-DRO | Alexnet · Imbalanced CIFAR-10 class 4 test | accuracy: 55.83 -> 58.65 | Helped |
| Use constrained SFK-DRO instead of penalized PAN-DRO | Alexnet · Imbalanced CIFAR-10 class 5 test | accuracy: 49.50 -> 49.36 | Hurt |
| Optimize the same dual objective with SFK-DRO instead of PGD | Cressie-Read constrained DRO · Imbalanced CIFAR-10 | convergence speed | Helped |
| Use DRO instead of ERM | Alexnet · Imbalanced CIFAR-10 training | training loss | Hurt |
| Use DRO instead of ERM | Alexnet · Imbalanced CIFAR-10 test | test loss | Helped |
Surprising findings
- The paper finds the dual variable lambda* is not necessarily bounded away from zero, so it avoids the lambda >= lambda_0 assumption by adding a regularizer while keeping the hard constraint.
Limitations
- Experiments are limited to a single imbalanced CIFAR-10 setup with AlexNet.
- Smoothed CVaR result stated as O(epsilon^{-9}) with only a sketch in the appendix.
- Requires bounded, Lipschitz, and smooth loss assumptions.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: per-class test accuracy cross-entropy training and test loss convergence curves
Baselines: PGD PAN-DRO ERM
Builds on: penalized DRO for phi-divergence Frank-Wolfe method PGD for constrained non-convex optimization smoothed CVaR DRO
Closest papers by meaning
- Revisiting Large-Scale Non-convex Distributionally Robust Optimization (ICLR 2025 · similarity 0.94)
- Non-convex Distributionally Robust Optimization: Non-asymptotic Analysis (NeurIPS 2021 · similarity 0.93)
- Distributionally Robust Optimization with Bias and Variance Reduction (ICLR 2024 · similarity 0.90)
- Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport (NeurIPS 2024 · similarity 0.89)
- Drago: Primal-Dual Coupled Variance Reduction for Faster Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.89)
- Wasserstein Distributionally Robust Optimization through the Lens of Structural Causal Models and Individual Fairness (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.