Research map / Reinforcement learning / MDPs and planning
Linear Mixture Distributionally Robust Markov Decision Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes linear mixture distributionally robust MDPs whose uncertainty sets are balls around the mixture weight parameter, gives a meta offline algorithm with double pessimism, and proves suboptimality bounds under TV, KL, and chi-square divergences.
Problem
Robust policy learning under off-dynamics shift in offline RL, where conventional (s,a)-rectangular uncertainty sets are overly conservative and the design of uncertainty sets relies on prior structure.
Approach
Define uncertainty sets as f-divergence balls around the mixture weighting parameter of a linear mixture model (rather than around each state-action transition kernel), so the set is a refined representation of dynamics shift when basis modes are known; combine with transition-targeted ridge regression and double pessimism.
Key results
- Theoretical suboptimality bounds of order d H^2 C / sqrt(K) for TV, d H^2 C e^{H/λ} / (ρ sqrt(K)) for KL, and d(sqrt(ρ)H^3+H^2) C / sqrt(K) for chi-square divergence.
- In the simulated experiments, robust variants DRTTR and DRVTR outperform non-robust TTR and VTR when perturbation is large, per the paper's reported average reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use robust DRTTR or DRVTR instead of the corresponding non-robust method | DRTTR and DRVTR · Simulated target environments with large perturbation | average reward | Helped |
| Use a linear mixture uncertainty set instead of an (s,a)-rectangular uncertainty set | Linear mixture DRMDP · TV-divergence uncertainty sets | uncertainty set conservativeness | Helped |
Limitations
- The meta-algorithm relies on planning oracles and is computationally intractable; practical algorithms rely on heuristic Nelder-Mead optimization and lack theoretical guarantees.
- Requires prior knowledge of basis modes and linear mixture structure; misspecification effects are not analyzed.
- Experiments are limited to a small simulated linear MDP instance.
Experimental setup
Datasets: simulated linear mixture MDP (5 states, 16 actions, H=3) with 500 offline trajectories
Benchmarks: simulated linear MDP instance adapted to linear mixture DRMDP
Metrics: average reward over 100 episodes suboptimality bound
Baselines: TTR VTR
Builds on: double pessimism linear mixture MDP (value-targeted regression) d-rectangular linear DRMDP (s,a)-rectangular DRMDP
Closest papers by meaning
- Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity (ICLR 2024 · similarity 0.88)
- Solving Robust Markov Decision Processes: Generic, Reliable, Efficient (AAAI 2025 · similarity 0.88)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.87)
- Online MDP with Prototypes Information: A Robust Adaptive Approach (AAAI 2025 · similarity 0.86)
- A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes (ICML 2024 · similarity 0.86)
- Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.