Research map / Reinforcement learning / Offline RL
Is Pessimism Provably Efficient for Offline RL?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a pointwise uncertainty penalty to the estimated Bellman update | PEVI for general MDPs · General episodic MDP | suboptimality: 2\sum_{h=1}^H\EE_{\pi^*}[\Gamma_h(s_h,a_h)\mid s_1=x] | Helped |
| Add pessimism using a valid uncertainty quantifier | PEVI · General MDP | spurious-correlation contribution to suboptimality | Helped |
| Use the linear-MDP confidence penalty with ridge regression | PEVI · Linear MDP | suboptimality: 2 \beta \sum_{h=1}^H\EE_{\pi^*}[ (\phi(s_h,a_h)^\top \Lambda_h^{-1}\phi(s_h,a_h))^{1/2} \mid s_1=x] | Helped |
| Ensure the dataset sufficiently covers the optimal-policy trajectory | PEVI · Linear MDP | suboptimality: c' \cdot d^{3/2} H^2 K^{-1/2} \sqrt{\zeta} | Helped |
| Use the rank-r condition on optimal-policy feature covariance | PEVI · Linear MDP | suboptimality: c''\cdot d H^2 K^{-1/2}\sqrt{\zeta} | Helped |
| Add reverse-ordered data splitting across value-iteration steps (ablation) | PEVI for linear function approximation · Linear MDP | suboptimality upper-bound scaling parameter: \tilde \cO(dH) -> \tilde \cO(\sqrt{d} H) | Helped |
| Split the dataset, reducing the sample available per step (ablation) | PEVI · Well-explored linear MDP | sample efficiency: \sqrt{H} | Mixed |
| Use a data-splitting kernel PEVI variant (ablation) | PEVI · RKHS with finite, exponential, or polynomial eigenvalue decay | suboptimality | Helped |
| Avoid requiring uniform dataset coverage or policy affinity constraints | PEVI · Linear MDP | assumptions required for the suboptimality guarantee | Helped |
| Focus the uncertainty bound on the optimal-policy trajectory rather than unrelated trajectories | PEVI · Linear MDP | suboptimality dependence on dataset coverage | Helped |
Closest papers by meaning
- Towards Instance-Optimal Offline Reinforcement Learning with Pessimism (NeurIPS 2021 · similarity 0.94)
- Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning (ICLR 2024 · similarity 0.94)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.93)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.93)
- Information-Directed Pessimism for Offline Reinforcement Learning (ICML 2024 · similarity 0.92)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.