Research map / Reinforcement learning / Offline RL
Information-Directed Pessimism for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes information-directed pessimism (IDP), an offline tabular RL penalty based on discrete Stein discrepancy (a one-sample test) that generalizes LCB pessimism to multimodal transitions, applied to value iteration and Q-learning.
Problem
Offline RL suffers from distribution mismatch between batch data and the current/optimal policy; standard LCB pessimism relies on unimodality assumptions that often fail and can be overly conservative.
Approach
Replace concentration-bound (LCB) penalties with a discrete Kernelized/Stein discrepancy computed in closed form over a Hamming kernel, which measures data-vs-true transition mismatch without two-sample tests and allows mixture-family transitions; the penalty is used in value iteration (IDP-VI) and asynchronous Q-learning (IDP-Q).
Key results
- IDP-VI achieves 59.74 return on Portfolio hard dataset versus 14.22 for VI-LCB (paper constants) and 58.44 for vanilla VI, approaching oracle VI at 69.0.
- On Random MDP random dataset, IDP-VI reaches 35.80 versus 31.04 for VI-LCB (tuned) and 2.14 for VI-LCB (paper constants) under oracle VI 37.03.
- Theory: IDP-VI has regret bound O(S C* / ((1-gamma)^4 T)) improving LCB by a factor 1/(1-gamma) and extends to mixture-family transitions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IDP-VI instead of VI Vanilla | Offline value iteration · Frozen Lake-Hard | return: 0.33 -> 0.52 | Helped |
| Use IDP-VI instead of VI-LCB (Td.) | Offline value iteration · Portfolio-Hard | return: 15.69 -> 59.74 | Helped |
| Use IDP-VI instead of VI-LCB (Td.) | Offline value iteration · DeepSea-Random | return: 2.27 -> 2.40 | Helped |
| Use IDP-VI instead of VI-LCB (Td.) | Offline value iteration · PriorMDP-Random | return: 12.93 -> 23.14 | Helped |
| Use IDP-VI instead of VI Vanilla | Offline value iteration · Random MDP-Random | return: 31.83 -> 35.80 | Helped |
| Use IDP-Q instead of QL | Offline Q-learning · Frozen Lake-Hard | return: 0.69 -> 0.73 | Helped |
| Use IDP-Q instead of QL | Offline Q-learning · Portfolio-Hard | return: 56.13 -> 56.29 | Helped |
| Use IDP-Q instead of CQL | Offline Q-learning · DeepSea-Random | return: 0.32 -> 0.03 | Hurt |
| Use IDP-Q instead of QL | Offline Q-learning · PriorMDP-Random | return: 15.81 -> 15.50 | Hurt |
| Use IDP-Q instead of QL | Offline Q-learning · Random MDP-Random | return: 23.03 -> 26.65 | Helped |
Surprising findings
- Reported LCB penalties with paper-specified constants can underperform even vanilla value iteration without any penalty (e.g., FrozenLake 0.01 vs 0.73 for VI-Vanilla... and PriorMDP-type cases), suggesting LCB is hard to tune in practice.
Limitations
- Stein penalty computation scales poorly with batch size (cubic per step), requiring coreset approximations.
- Analysis restricted to finite discrete tabular state and action spaces.
- Score-function estimation from offline data relies on optimistic transition estimates; the theory uses true score when available.
Experimental setup
Datasets: Portfolio Optimization Frozen Lake DeepSea PriorMDP Random MDP
Benchmarks: Portfolio Optimization Frozen Lake DeepSea PriorMDP Random MDP
Metrics: cumulative test return
Baselines: VI-LCB (Rashidinejad et al., 2021) VI Vanilla Q-learning Q-LCB (Shi et al., 2022) CQL (Kumar et al., 2020) VI with oracle model
Builds on: Pessimistic value iteration (VI-LCB) Asynchronous Q-learning with LCB Information-directed sampling Discrete Stein discrepancy (Yang et al., 2018) Conservative Q-learning
Closest papers by meaning
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.92)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Worst-Case Offline Reinforcement Learning with Arbitrary Data Support (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.