Research map / Reinforcement learning
Offline RL: research map
646 papers on Offline RL in Reinforcement learning, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 8 approaches. The busiest year so far is 2026.
What it is
Offline RL learns a policy entirely from a fixed dataset of past interactions, without trying new actions, which matters when exploration is costly or unsafe (healthcare, robots). The core difficulty is that the policy may choose actions the data never covered, so methods stay conservative.
Terms you will see on this page:
- Offline RL
- learning a policy from logged data only.
- Distribution shift (in RL)
- the learned policy visiting states and actions the dataset lacks.
- Conservatism
- penalising values of actions that the data does not support.
Start reading with the most cited papers here:
- Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations)
- Decision Transformer: Reinforcement Learning via Sequence Modeling (NeurIPS 2021 · 464 citations)
Within Reinforcement learning, its share shrank from 15.3% in 2023–24 to 10.7% in 2025–26 (240 → 247 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Offline RL in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Offline RL sample complexity theory rising: 22% → 27% of the topic (53 → 66 papers)
- Diffusion models for offline decision making rising: 12% → 17% of the topic (29 → 41 papers)
- Behavior regularization for offline RL rising: 5% → 8% of the topic (12 → 19 papers)
- Offline-to-online fine-tuning falling: 20% → 15% of the topic (47 → 36 papers)
- Pessimism and uncertainty in offline RL falling: 11% → 7% of the topic (27 → 17 papers)
Papers from 2025–2026 that are already cited most:
- SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · 4 citations)
Approaches and key papers
Offline RL algorithms · 409 papers
Keywords: offline reinforcement, offline rl, conservative
Approaches in this cluster:
- Offline RL sample complexity theory (159 papers)
Prove sample-complexity bounds for offline RL under limited data coverage. - Simple practical offline RL (148 papers)
Improve offline RL with minimalist algorithms, reweighting, and symmetry-based data efficiency. - Pessimism and uncertainty in offline RL (65 papers)
Penalize uncertain values with ensembles, Bayesian models, and pessimistic value iteration. - Behavior regularization for offline RL (37 papers)
Constrain learned policies toward the dataset with divergence penalties and in-sample learning.
Most cited and most cited since 2024:
- Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020 · 556 citations)
- MOPO: Model-based Offline Policy Optimization (NeurIPS 2020 · 238 citations)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · 7 citations)
- Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning (NeurIPS 2024 · 7 citations)
Offline-to-online and sequence models · 237 papers
Keywords: offline online, transformer, online reinforcement
Approaches in this cluster:
- Offline-to-online fine-tuning (99 papers)
Pretrain on offline data then fine-tune online with calibration and balanced exploration. - Diffusion models for offline decision making (87 papers)
Use diffusion sampling and representation learning for offline and offline-to-online decisions. - Decision Transformer sequence models (43 papers)
Cast offline RL as sequence modeling with transformers, enhanced by value guidance. - Offline policy evaluation and selection (8 papers)
Evaluate and select policies from offline data using augmentation and re-weighting.
Most cited and most cited since 2024:
- Decision Transformer: Reinforcement Learning via Sequence Modeling (NeurIPS 2021 · 464 citations)
- A General Offline Reinforcement Learning Framework for Interactive Recommendation (AAAI 2021 · 63 citations)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · 19 citations)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · 19 citations)
Related topics in Reinforcement learning
- MDPs and planning (726)
- Visual and representation RL (2,264)
- Policy gradient and actor-critic (1,570)
- Imitation learning (450)
- RL from feedback (758)
