Research map / Reinforcement learning / MDPs and planning
Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes d-rectangular linear robust regularized MDPs (d-RRMDP) and the R2PVI offline algorithm with TV, KL and chi-square f-divergence regularization, giving closed-form duals, instance-dependent suboptimality bounds and a near-matching lower bound.
Problem
Offline robust RL under dynamics shift with uncertainty sets that are unstructured or conservative, and d-rectangular DRMDPs whose dual oracles are computationally costly and theoretically limited to TV divergence.
Approach
Replace the d-rectangular uncertainty set with a divergence penalty on each latent factor (d-RRMDP), so the robust Bellman dual has closed form for TV and KL and a one-dimensional variational form for chi-square, enabling a pessimistic value iteration with linear function approximation.
Key results
- R2PVI (TV, KL, chi-square) gives suboptimality bounds matching P2MPO order d^2 H^2 K^{-1/2} for TV, and the first chi-square d-rectangular bound.
- Information-theoretic lower bound shows the instance-dependent term is intrinsic up to a factor beta.
- On American put option, R2PVI computation cost is comparable to PEVI and lower than DRPVI and DRVI-L, which need dual optimization oracles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use R2PVI instead of PEVI | offline simulated linear MDP · small dynamics perturbation | average reward | Hurt |
| use R2PVI instead of PEVI | offline simulated linear MDP · significant dynamics shift | average reward | Helped |
| decrease the regularizer lambda | R2PVI · simulated linear MDP with dynamics shifts | robustness | Helped |
| use R2PVI instead of DRPVI or DRVI-L | offline simulated American Put Option · computation time | computation time | Helped |
| use R2PVI instead of PEVI | offline simulated American Put Option · computation time | computation time | No clear effect |
| choose appropriate regularizer lambda and uncertainty level rho | R2PVI and DRPVI · simulated American Put Option | performance | No clear effect |
| use R2PVI-TV instead of P2MPO (TV) | d-RRMDP versus d-DRMDP · TV, regularized partial coverage versus robust partial coverage | suboptimality gap upper bound: \tilde{O}(d^{2}H^2 K^{-1/2}) | No clear effect |
Limitations
- Upper bounds are not tight relative to the lower bound; the gap is left open
- Instance-independent bounds use an assumed regularized partial coverage condition
- Experiments are small simulated environments run on a laptop CPU
Experimental setup
Datasets: simulated linear MDP (Liu and Xu 2024 style) simulated American put option
Benchmarks: simulated linear MDP simulated American put option
Metrics: average reward computation time suboptimality gap
Baselines: PEVI DRPVI DRVI-L
Builds on: PEVI DRPVI DRVI-L RRMDP d-rectangular linear DRMDP
Closest papers by meaning
- Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data (ICML 2024 · similarity 0.93)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.90)
- Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization (AAAI 2024 · similarity 0.90)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.