Research map / Reinforcement learning / Policy gradient and actor-critic
Distributionally Robust Policy Learning under Concept Drifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a doubly-robust estimator and a KL-DRO policy learning algorithm for robust offline policy learning when only the conditional outcome distribution shifts (concept drift), with a Hamming-entropy-integral regret bound of order n^{-1/2} and a matching lower bound.
Problem
Offline policy learning that is robust to worst-case distribution shift in the conditional outcome distribution Y|X (concept drift) with covariate distribution fixed, avoiding the over-conservatism of joint-shift DRO methods and handling general covariate spaces.
Approach
Separate the distribution shift into a KL-bounded conditional outcome shift, derive a dual (ERM-form) representation of the worst-case value per covariate, and debias it with a doubly-robust estimator that stays root-n consistent even when nuisance optimizers converge slowly; policy learning maximizes the estimated value over a policy class, with per-action ERM to avoid per-policy nuisance fitting
Key results
- On simulated data with delta=0.05, the proposed policy has robust value about 0.227-0.231 versus 0.055-0.066 for SNLN across n=7500 to 19500 (50 seeds).
- At delta=0.2 the proposed policy achieves 0.078-0.083 versus 0.018-0.022 for SNLN.
- Worst-case reward on perturbed KL-sphere test sets at delta=0.1 is 0.207-0.217 for the proposed method versus 0.188-0.202 for SNLN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use concept-drift robust policy learning rather than the SNLN joint-shift benchmark | Policy learning on the simulated dataset, δ=0.05 · Simulated dataset | robust policy value: 0.0664±0.003 -> 0.2310±0.001 | Helped |
| Use concept-drift robust policy learning rather than the SNLN joint-shift benchmark | Policy learning on the simulated dataset, δ=0.1 · Simulated dataset | robust policy value: 0.0616±0.004 -> 0.1678±0.002 | Helped |
| Use concept-drift robust policy learning rather than the SNLN joint-shift benchmark | Policy learning on the simulated dataset, δ=0.2 · Simulated dataset | robust policy value: 0.0219±0.003 -> 0.0831±0.002 | Helped |
| Use concept-drift robust policy learning rather than the SNLN joint-shift benchmark | Policy learning under concept drift, worst-case KL-sphere test · Simulated dataset, δ=0.1 | worst-case policy reward: 0.2020±0.004 -> 0.2167±0.003 | Helped |
| Increase the uncertainty radius δ | Algorithm LN and SNLN · Simulated dataset | robust policy value | Hurt |
Limitations
- Choice of the robustness radius delta strongly affects performance and has no data-driven selection in the paper.
- Extension to continuous action spaces is left open.
- Requires assumptions on smoothness and convergence rates of nuisance optimizers (slower-than-root-n rates for some nuisances).
Experimental setup
Datasets: simulated linear-boundary Gaussian reward data French unemployment assistance randomized experiment dataset
Benchmarks: simulated linear boundary dataset French unemployment assistance experiment
Metrics: empirical robust policy value worst-case policy reward on KL-sphere perturbed test sets
Baselines: SNLN
Builds on: SNLN (Si et al. distributionally robust policy learning) double machine learning policytree
Closest papers by meaning
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.87)
- Distributionally Robust Policy Evaluation and Learning for Continuous Treatment with Observational Data (AAAI 2025 · similarity 0.87)
- Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes (NeurIPS 2024 · similarity 0.86)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.86)
- Achievable distributional robustness when the robust risk is only partially identified (NeurIPS 2024 · similarity 0.86)
- Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.