Research map / Reinforcement learning / Offline RL
Pessimism and uncertainty in offline RL
Penalize uncertain values with ensembles, Bayesian models, and pessimistic value iteration.
59 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Offline RL, keyword cluster “offline reinforcement · offline rl · conservative”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
27 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 11.2% to 6.9%.
Open questions the data points to
- Papers report CQL on D4RL hopper-medium (normalized score) anywhere from 58 to 86.6: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · COMBO: Conservative Offline Model-Based Policy Optimization (NeurIPS 2021 · 51 citations)
- 2022 · Constraints Penalized Q-learning for Safe Offline Reinforcement Learning (AAAI 2022 · 51 citations)
- 2023 · When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning (ICLR 2023 · 4 citations)
- 2024 · Offline Model-Based Optimization via Policy-Guided Gradient Search (AAAI 2024 · 6 citations)
- 2025 · SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · 4 citations)
- 2026 · VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning (ICML 2026 · 0 citations)
Earliest papers here
- COMBO: Conservative Offline Model-Based Policy Optimization (NeurIPS 2021 · 51 citations)
- Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble (NeurIPS 2021 · 48 citations)
- What are the Statistical Limits of Offline RL with Linear Function Approximation? (ICLR 2021 · 35 citations)
What has been tried
47 papers here report 426 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 11%, hurt 20%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use offline RL rather than standard RL to train the policy | Superconductor · normalized objective | 0.528 ± 0.012 -> 0.563 ± 0.058 | Helped | Offline Model-Based Optimization via Policy-Guided Gradient Search (AAAI 2024 · 6 citations) |
| Include action and next state in the search vector (s⊕a⊕s′ instead of s⊕s′) | D4RL MuJoCo (6 selected datasets) · Spearman rank correlation (ρ) | 0.56 -> 0.83 | Helped | SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · 4 citations) |
| Use O-MDP instead of P-MDP for rollout and policy optimization (OROO) | reported offline RL dataset · real-environment return | No clear effect | Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · 3 citations) | |
| Add entropy-based uncertainty weighting to TRACER | MuJoCo simultaneous corruptions · average score | Helped | Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions (NeurIPS 2024 · 3 citations) | |
| Add the SQL trick to SQL | Gym-Mujoco-total · normalized score | 668.8 -> 703.2 | Helped | Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · 2 citations) |
| Add the guardian to CQL | MIMIC-III sepsis treatment · MCR (10^-3) | 789 ± 5.64 -> 909 ± 2.52 | Helped | Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies (NeurIPS 2025 · 1 citation) |
| use λ-returns instead of H-step returns for critic and policy updates | AntMaze total · success rate total | 290.3 -> 461.8 | Helped | Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning (ICLR 2025 · 1 citation) |
| Add reward relabeling to discount-only rank blending | 27 datasets, average · relative improvement | -0.02 -> 0.02 | Helped | Improving Offline RL by Blending Heuristics (ICLR 2024 · 1 citation) |
| Add prioritized dataset (PD) | HalfCheetah-medium · normalized average return | Helped | Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · 1 citation) | |
| Add dynamic uncertainty estimation | Walker2d-medium-v2 · normalized score | Helped | Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · 0 citations) | |
| Remove conservative estimation of reward Q-values | PointGoal · Reward | Hurt | POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning (CVPR 2024 · 0 citations) | |
| Set the variance floor to the extreme value so all regression targets receive the same weights | Linear MDP offline RL · suboptimality upper bound | No clear effect | Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| CQL | D4RL hopper-medium | normalized score | 58 – 86.6 | 2 papers, e.g. NeurIPS 2021 and NeurIPS 2021 |
| MOPO | HalfCheetah-jump | average return | 4017 – 4412 | 2 papers, e.g. NeurIPS 2021 and AAAI 2024 |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- D4RL hopper · 2 of 11 papers
- D4RL antmaze · 2 of 11 papers
Baselines
- CQL · 3 of 11 papers
- EDAC · 3 of 11 papers
- TD3-BC · 2 of 11 papers
- IQL · 2 of 11 papers
- MOPO · 2 of 11 papers
- MOBILE · 2 of 11 papers
- COMBO · 2 of 11 papers
- RAMBO · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 44 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- University of Science and Technology of China · 4 papers
- JD Explore Academy · 2 papers
- Indian Institute of Technology · 2 papers
- Tongji University · 2 papers
- Google · 2 papers
- University of California, Berkeley · 2 papers
- Microsoft · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Offline RL sample complexity theory (156 papers)
- Simple practical offline RL (144 papers)
- Behavior regularization for offline RL (35 papers)
Elsewhere in the topic
- Offline-to-online fine-tuning (99 papers)
- Diffusion models for offline decision making (82 papers)
- Decision Transformer sequence models (40 papers)
- Offline policy evaluation and selection (8 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).