Research map / Reinforcement learning / Offline RL
A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a single distributionally robust MDP uncertainty set whose radius adds a data-sparsity penalty to the model-mismatch radius, giving offline RL sample-complexity guarantees under TV, chi-square and KL uncertainty models.
Problem
Offline RL when the dataset-collecting environment differs from the deployment environment (model mismatch) and the dataset covers state-action pairs only partially, so the nominal model is estimated with error.
Approach
Combine the LCB-style pessimism for limited data and DRO-style pessimism for model mismatch into one (s,a)-rectangular uncertainty set around the empirical kernel, with radius R + kappa(s,a), where kappa shrinks with visit counts; a value-function-based radius design is used for non-metric divergences (chi-square, KL).
Key results
- Theoretical sample complexity under TV improves on prior bound of [5]: O~(S Cπ* / ((1-γ)^3 ε^2) · min{S/(1-γ), 1/μmin}) vs. [5]'s S^2 dependence.
- Under KL divergence, the bound O~(Cπ* S /((1-γ)^4 P_min ε^2)) matches [36]; under chi-square, the paper gives the first offline robust result with O~(Cπ* S/((1-γ)^4 ε^2)) plus burn-in.
- In experiments, the value-function-based construction has smaller sub-optimality gap and converges faster than the distribution-based construction and DRVI-LCB; the KL-based method runs faster than LCB in execution time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the value function-based radius instead of the distribution-based radius (ablation) | Unified DRO · Frozen-Lake and Gambler, TV uncertainty set | sub-optimality gap | Helped |
| Use the value function-based radius instead of the distribution-based radius (ablation) | Unified DRO · Frozen-Lake and Gambler, TV uncertainty set | convergence rate | Helped |
| Use the value function-based radius instead of the distribution-based radius (ablation) | Unified DRO · TV uncertainty set | sample complexity: \tilde{O}\left(\frac{C^{\pi^*}S}{(1-\gamma)^3\epsilon^2}\min\left\{\frac{S}{1-\gamma},\frac{1}{\mu_{\min}}\right\}\right | Helped |
| Use a unified DRO uncertainty set rather than the method in [5] | Offline robust RL · TV uncertainty set | sample complexity: \tilde{O}\left(\frac{C^{\pi^*}S}{(1-\gamma)^3\epsilon^2}\min\left\{\frac{S}{1-\gamma},\frac{1}{\mu_{\min}}\right\}\right | Helped |
| Use a unified DRO uncertainty set rather than [36] | Offline robust RL · KL divergence uncertainty set | sample complexity: \tilde{O}\left(\frac{C^{\pi^*}S}{(1-\gamma)^4\epsilon^2P_{\min}}\right) | No clear effect |
| Use a unified DRO uncertainty set rather than the method in [5] | Offline robust RL · KL divergence uncertainty set | sample complexity: \tilde{O}\left(\frac{C^{\pi^*}S}{(1-\gamma)^4\epsilon^2P_{\min}}\right) versus \tilde{O}\left(\frac{C^{\pi^*}_{\mathrm{r | Helped |
| Use the proposed method rather than DRVI-LCB | Offline robust RL · Frozen-Lake and Gambler, χ2 divergence uncertainty set | convergence rate | No clear effect |
| Use the proposed method rather than non-robust DP | Offline robust RL · Frozen-Lake and Gambler, χ2 divergence uncertainty set | convergence rate | Helped |
| Use the proposed method rather than DRVI-LCB | Offline robust RL · Frozen-Lake and Gambler, KL divergence uncertainty set | convergence rate | No clear effect |
| Use the proposed method rather than non-robust DP | Offline robust RL · Frozen-Lake and Gambler, KL divergence uncertainty set | convergence rate | Helped |
Surprising findings
- The distribution-based uncertainty set under TV gives an overly conservative policy, and a value-function-based radius design performs better empirically and theoretically.
Limitations
- Analysis is tabular; extension to linear MDPs and function approximation is left to future work.
- Radius R is assumed pre-set by domain experts and not learned.
Experimental setup
Datasets: synthetic offline datasets generated on Frozen-Lake and Gambler MDPs
Benchmarks: Frozen-Lake Gambler problem
Metrics: sub-optimality gap execution time
Baselines: DRVI-LCB Non-robust dynamic programming distribution-based construction (own variant) Shi & Chi 2022 KL model Blanchet et al. 2023 double pessimism TV/KL
Builds on: distributionally robust MDP (Iyengar 2005 / Nilim & El Ghaoui 2005) LCB pessimism for offline RL (Xie et al. 2021) DRVI-LCB (Shi & Chi 2022) robust single-policy clipped concentrability
Closest papers by meaning
- Information-Directed Pessimism for Offline Reinforcement Learning (ICML 2024 · similarity 0.92)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.92)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.91)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.