Research map / Reinforcement learning / Offline RL
Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes computationally efficient offline algorithms (DRPVI and variance-aware VA-DRPVI) for distributionally robust offline RL under d-rectangular linear DRMDPs with TV uncertainty, with instance-dependent suboptimality bounds and a matching information-theoretic lower bound.
Problem
Robust offline RL with function approximation under dynamics uncertainty: nonlinearity and computational burden of the robust Bellman operator, and distribution shift between nominal data and perturbed models.
Approach
Uses d-rectangular linear DRMDPs with factor-wise TV uncertainty sets so the robust Bellman update decouples into d separate ridge regressions; adds variance-weighted regression exploiting range shrinkage of robust value functions to get sharper, near-minimax bounds.
Key results
- VA-DRPVI achieves suboptimality bound with sup over uncertainty set and variance-weighted covariance, improving over DRPVI by a factor of H when rho=O(1).
- Information-theoretic lower bound matches the VA-DRPVI upper bound up to a sqrt(d) factor (beta_2 = tilde O(sqrt d)).
- In simulations, DRPVI and VA-DRPVI are more robust to environment perturbation than non-robust PEVI, with VA-DRPVI slightly better in most settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| incorporate variance information into the function approximation mechanism (ablation) | DRPVI · d-rectangular linear DRMDPs | instance-dependent upper bound on suboptimality gap: $\bSigma_h^{\star-1} \preceq H^2\bLambda_h^{-1}$ | Helped |
| use the range-shrinkage property for variance-aware analysis | VA-DRPVI · d-rectangular linear DRMDPs with $\rho=O(1)$ | instance-dependent upper bound on suboptimality gap: by an order of $H$ | Helped |
| use variance information and range shrinkage instead of the DRPVI bound | VA-DRPVI versus DRPVI · simulated linear DRMDP | policy performance | Helped |
| use robust training rather than non-robust PEVI | DRPVI and VA-DRPVI · simulated linear DRMDP under environmental perturbation | policy robustness | Helped |
| estimate variance of the untruncated robust value instead of separate truncated values for each α (ablation) | VA-DRPVI · d-rectangular linear DRMDPs | computation cost | Helped |
| estimate variance of the untruncated robust value instead of separate truncated values for each α (ablation) | VA-DRPVI · d-rectangular linear DRMDPs | instance-dependent upper bound on suboptimality gap | No clear effect |
Surprising findings
- Robust offline RL with function approximation appears probably harder than standard offline RL, requiring worst-case coverage over the whole uncertainty set rather than nominal coverage alone.
Limitations
- Restricted to d-rectangular linear DRMDPs with TV uncertainty sets and shared state-action spaces between nominal and perturbed models.
- Bounds carry a tilde O(sqrt d) or sqrt(d)H gap relative to the lower bound.
- Experiments are small simulated linear MDPs with only three steps and five states.
Experimental setup
Datasets: simulated linear DRMDP offline dataset
Benchmarks: simulated linear DRMDP environment
Metrics: policy performance under perturbation level
Baselines: PEVI
Builds on: pessimistic value iteration PEVI variance-weighted ridge regression reference-advantage decomposition d-rectangular DRMDP
Closest papers by meaning
- Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.92)
- Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning (ICLR 2024 · similarity 0.92)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.91)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.91)
- Worst-Case Offline Reinforcement Learning with Arbitrary Data Support (NeurIPS 2024 · similarity 0.91)
- Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.