Research map / Reinforcement learning / Offline RL
Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes offline risk-sensitive pessimistic value iteration algorithms (RSPVI and variance-aware VA-RSPVI) for linear MDPs under the entropic risk measure, with suboptimality bounds that recover risk-neutral rates as beta goes to 0.
Problem
Provably sample-efficient offline reinforcement learning under the entropic risk measure, which had only been studied online and mostly in tabular MDPs.
Approach
Uses a shifting-and-scaling transformation of the exponential value function so regression targets share a common range, combined with pessimistic bonuses in the exponential Bellman equation; the variance-aware variant adds weighted ridge regression with variance estimates from an auxiliary dataset and reference-advantage decomposition.
Key results
- RSPVI achieves suboptimality O~(d) * (e^{|beta|H}-1)/|beta| * sum_h E[||phi||], with no coverage assumption.
- Under data coverage, the dimension dependence improves from d to sqrt(d) via reference-advantage decomposition.
- The variance-aware VA-RSPVI further sharpens dependence on the risk-sensitivity factor, and recovers risk-neutral rates as beta goes to 0+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a data coverage assumption and adjust the uncertainty bonus | RSPVI · Linear MDP offline RL | suboptimality upper bound: \tilde O(d) -> \tilde O(\sqrt{d}) | Helped |
| Add variance estimation and variance-weighted regression | RSPVI · Linear MDP offline RL with data coverage | suboptimality upper bound: \tilde O\big(\sqrt{d}\big)\frac{e^{\abs{\beta}H}-1}{\abs{\beta}} -> \tilde O\big(\sqrt{d}\big)\frac{1}{\abs{\beta}} | Helped |
| Set the variance floor to the extreme value so all regression targets receive the same weights (ablation) | VA-RSPVI · Linear MDP offline RL | suboptimality upper bound | No clear effect |
Limitations
- Results are purely theoretical with no empirical experiments.
- Variance-aware algorithm requires an auxiliary dataset (or data splitting) and coverage assumptions (minimum eigenvalue kappa) for sharper bounds.
- Extension to low-rank MDPs and kernel settings left as future work.
Experimental setup
Metrics: suboptimality bound (theoretical)
Baselines: risk-neutral pessimistic offline RL bounds (Xiong et al. 2022 style) online risk-sensitive linear MDP algorithm (Fei et al.-type analysis)
Builds on: pessimistic value iteration for linear MDPs exponential Bellman equation for entropic risk variance-aware offline RL (risk-neutral) reference-advantage decomposition
Closest papers by meaning
- A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.93)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.91)
- Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.91)
- Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.90)
- Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents (NeurIPS 2025 · similarity 0.90)
- Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.