Research map / Reinforcement learning / MDPs and planning
Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an empirical-variance Bernstein penalty instead of a uniform O(H/√n_{s_h,a_h}) penalty (ablation) | Pessimistic value iteration · Finite-horizon tabular MDPs; optimal policy covered by behavior policy | suboptimality gap: C'H\sum_{h=1}^H\sum_{(s_h,a_h)\in\mathcal{C}_h}d^{\pi^\star}_h(s_h,a_h)\cdot\sqrt{\frac{\iota}{ n\cdot d^\mu_h{(s_h,a_h) | Helped |
| Use a pessimistic augmented MDP that maps unsupported state-action pairs to a zero-reward absorbing state | APVI · Finite-horizon tabular MDPs with arbitrary behavior policy | suboptimality gap: The paper gives the resulting bound as \sum_{h=2}^{H+1}d^{\dagger\pi^\star}_h(s^\dagger_h) plus the covered-region varia | Helped |
| Use APVI's instance-dependent variance bound under uniform data coverage | APVI · Uniform data coverage | sample complexity: The APVI bound implies \widetilde{O}(H^3/d_m\epsilon^2), matching the stated optimal rate. | Helped |
| Adapt the pessimistic penalty to transition and reward variance | APVI · Fully deterministic systems | suboptimality gap: \frac{H^3}{n\bar{d}_m}; the paper contrasts this with existing non-adaptive results of order \frac{1}{\sqrt{n}}. | Helped |
Closest papers by meaning
- Worst-Case Offline Reinforcement Learning with Arbitrary Data Support (NeurIPS 2024 · similarity 0.93)
- A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs (ICML 2024 · similarity 0.92)
- Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning (ICLR 2024 · similarity 0.92)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.91)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Information-Directed Pessimism for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.