Research map / Reinforcement learning / Offline RL
Simple practical offline RL
Improve offline RL with minimalist algorithms, reweighting, and symmetry-based data efficiency.
144 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Offline RL, keyword cluster “offline reinforcement · offline rl · conservative”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
48 papers in 2023–24 and 53 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 20.0% to 21.5%.
Open questions the data points to
- Papers report IQL on antmaze-large-play (normalized score) anywhere from 39.6 to 55.7: comparisons against it depend on whose number you take.
- Papers report IQL on antmaze-medium-diverse (normalized score) anywhere from 70 to 82.3: comparisons against it depend on whose number you take.
No longer reported by papers here: antmaze-large-play (normalized score, last in 2024); antmaze-large-diverse (normalized score, last in 2023); antmaze-medium-diverse (normalized score, last in 2023); antmaze-medium-play (normalized score, last in 2023); antmaze-umaze-diverse (normalized score, last in 2023); Hopper-medium (normalized score, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Offline Evaluation of Online Reinforcement Learning Algorithms (AAAI 2016 · 12 citations)
- 2019 · Surveys without Questions: A Reinforcement Learning Approach (AAAI 2019 · 1 citation)
- 2020 · Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations)
- 2021 · A Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2021 · 162 citations)
- 2022 · Offline Reinforcement Learning with Implicit Q-Learning (ICLR 2022 · 130 citations)
- 2023 · Weighted Policy Constraints for Offline Reinforcement Learning (AAAI 2023 · 12 citations)
- 2024 · Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · 7 citations)
- 2025 · Data Center Cooling System Optimization Using Offline Reinforcement Learning (ICLR 2025 · 7 citations)
- 2026 · Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies (AAAI 2026 · 2 citations)
Earliest papers here
- Offline Evaluation of Online Reinforcement Learning Algorithms (AAAI 2016 · 12 citations)
- Surveys without Questions: A Reinforcement Learning Approach (AAAI 2019 · 1 citation)
- Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
antmaze-large-play · normalized score (higher is better)
6 papers, 2020–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 15.8 | Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations) |
| 2022 | 64.4 | DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning (NeurIPS 2022 · 0 citations) |
| 2023 | 67.3 | Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning (ICLR 2023 · 0 citations) |
antmaze-large-diverse · normalized score (higher is better)
5 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 14.9 | Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations) |
| 2022 | 74.1 | DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning (NeurIPS 2022 · 0 citations) |
| 2023 | 74.3 | Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning (ICLR 2023 · 0 citations) |
antmaze-medium-diverse · normalized score (higher is better)
5 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 53.7 | Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations) |
| 2022 | 87.1 | DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning (NeurIPS 2022 · 0 citations) |
antmaze-medium-play · normalized score (higher is better)
5 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 61.2 | Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations) |
| 2022 | 92.3 | DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning (NeurIPS 2022 · 0 citations) |
antmaze-umaze-diverse · normalized score (higher is better)
5 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 84 | Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations) |
| 2022 | 89 | DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning (NeurIPS 2022 · 0 citations) |
| 2023 | 94 | Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning (ICLR 2023 · 0 citations) |
Hopper-medium · normalized score (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 44.1 | Offline Model-based Adaptable Policy Learning (NeurIPS 2021 · 4 citations) |
| 2022 | 70.2 | Bootstrapped Transformer for Offline Reinforcement Learning (NeurIPS 2022 · 13 citations) |
| 2024 | 101.6 | The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · 1 citation) |
What has been tried
100 papers here report 1,042 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 10%, hurt 23%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add GNN blocks that encode spatial and control dependencies | Server Room B historical data · multi-step prediction error (MSE) | Helped | Data Center Cooling System Optimization Using Offline Reinforcement Learning (ICLR 2025 · 7 citations) | |
| Remove supported policy regularization | Cross-domain offline RL tasks · performance | Hurt | Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · 7 citations) | |
| Add Gaussian action noise without conservative constraints | walker2d-medium-replay-v2 · score | Hurt | OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · 4 citations) | |
| Add SCAS regularization to ordinary off-policy RL | D4RL · learned Q-value stability | Helped | Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression (NeurIPS 2024 · 4 citations) | |
| Use temporal abstraction with increasing sequence horizon H | kitchen-mixed-v0 · D4RL score | Mixed | Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · 2 citations) | |
| Use priority-weighted actor federation | Hopper · server performance | Helped | Federated Ensemble-Directed Offline Reinforcement Learning (NeurIPS 2024 · 2 citations) | |
| Use the diffusion model with DCM fusion rather than the diffusion model alone | no access to past offsets · normalized score | +97.8% relative performance improvement | Helped | Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · 1 citation) |
| use a nonlinear rather than linear value function | Gym - Locomotion · TD3-normalized performance | -1.17 | Helped | Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · 1 citation) |
| Set diversity regularizer eta to 10.0 instead of 1.0 | HalfCheetah-medium · normalized score | 78.7± 2.0 -> 74.8± 1.5 | Hurt | The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · 1 citation) |
| Increase synthetic data size from 16 to 256 | D4RL, Halfcheetah M-R · normalized return | 6.9 -> 35.9 | Helped | Offline Behavior Distillation (NeurIPS 2024 · 1 citation) |
| replace single-step RTG estimates with multi-step value estimates | Stochastic Taxi · episode return | -99 -> -57 | Helped | A Tractable Inference Perspective of Offline RL (NeurIPS 2024 · 1 citation) |
| Add TempDATA augmentation rather than naive model-based rollouts | D4RL AntMaze medium-play, medium-diverse, large-play, large-diverse · average return | Helped | Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| IQL | antmaze-large-play | normalized score | 39.6 – 55.7 | 4 papers, e.g. AAAI 2023 and NeurIPS 2022 |
| IQL | antmaze-medium-diverse | normalized score | 70 – 82.3 | 3 papers, e.g. AAAI 2023 and NeurIPS 2022 |
| IQL | antmaze-medium-play | normalized score | 71.2 – 82.7 | 3 papers, e.g. AAAI 2023 and NeurIPS 2022 |
| IQL | antmaze-large-diverse | normalized score | 47.5 – 50 | 3 papers, e.g. AAAI 2023 and NeurIPS 2022 |
| CQL | D4RL Adroit average (12 datasets) | normalized score | 14.6 – 40.3 | 2 papers, e.g. NeurIPS 2022 and NeurIPS 2023 |
| MOPO | Hopper-random | normalized score | 11.7 – 31.4 | 2 papers, e.g. NeurIPS 2021 and NeurIPS 2025 |
What reviewers will expect
Datasets and baselines named in the 30 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- D4RL · 4 of 30 papers
- D4RL AntMaze · 3 of 30 papers
- D4RL Kitchen · 3 of 30 papers
- D4RL MuJoCo medium-replay-v2 · 2 of 30 papers
- D4RL Adroit expert-v0 · 2 of 30 papers
Baselines
- IQL · 8 of 30 papers
- CQL · 5 of 30 papers
- MOPO · 3 of 30 papers
- HIQL · 3 of 30 papers
- DreamerV3 · 3 of 30 papers
- FQL · 3 of 30 papers
- TD3BC · 3 of 30 papers
- BC · 3 of 30 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 101 papers; from OpenAlex and the papers' affiliations).
- University of California, Berkeley · 7 papers
- Microsoft · 6 papers
- Carnegie Mellon University · 6 papers
- Tsinghua University · 5 papers
- University of Oxford · 5 papers
- Google · 4 papers
- The Chinese University of Hong Kong · 4 papers
- Zhejiang University · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Peking University · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Offline RL sample complexity theory (156 papers)
- Pessimism and uncertainty in offline RL (59 papers)
- Behavior regularization for offline RL (35 papers)
Elsewhere in the topic
- Offline-to-online fine-tuning (99 papers)
- Diffusion models for offline decision making (82 papers)
- Decision Transformer sequence models (40 papers)
- Offline policy evaluation and selection (8 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).