Research map / Reinforcement learning / Offline RL
Offline-to-online fine-tuning
Pretrain on offline data then fine-tune online with calibration and balanced exploration.
99 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Offline RL, keyword cluster “offline online · transformer · online reinforcement”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 1 tracked benchmark.
47 papers in 2023–24 and 36 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 19.6% to 14.6%.
Open questions the data points to
- BulletGym average (normalized cost): best went from 0.1 (up to 2024) to 0.08 in 2025 — Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning (ICML 2025)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Offline Reinforcement Learning as One Big Sequence Modeling Problem (NeurIPS 2021 · 41 citations)
- 2022 · Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration (ICLR 2022 · 27 citations)
- 2023 · Adaptive Policy Learning for Offline-to-Online Reinforcement Learning (AAAI 2023 · 24 citations)
- 2024 · A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · 19 citations)
- 2025 · Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · 7 citations)
- 2026 · Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models (AAAI 2026 · 1 citation)
Earliest papers here
- Offline Reinforcement Learning as One Big Sequence Modeling Problem (NeurIPS 2021 · 41 citations)
- EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (ICML 2021 · 24 citations)
- Representation Matters: Offline Pretraining for Sequential Decision Making (ICML 2021 · 22 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
BulletGym average · normalized cost (lower is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 0.1 | Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · 3 citations) |
| 2025 | 0.08 | Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning (ICML 2025) |
What has been tried
76 papers here report 809 measured changes (a component added, removed or swapped, compared with a baseline): helped 50%, no clear effect 11%, hurt 31%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add target-action perturbation (σ=0.3) rather than no noise (σ=0.0) | D4RL MuJoCo, Walker2d, Average · normalized average return | 76.5 ±19.1 → 95.4 ±4.7 | Helped | A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · 19 citations) |
| use shared policy representation instead of independently trained agents | CarGoal1 · normalized reward | 0.00 to 0.28 | Helped | Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · 7 citations) |
| Change the target value across tested values | Maze2D Umaze · normalized score | 142.1 ±5.8 (v=0.0125); 134.7 ±6.9 (v=0.025); 142.1 ±3.4 (v=0.05); 149.1 ±5.4 (v=0.1); 144.6 ±3.4 (v=0.2); 150.3 ±7.4 (v= | Mixed | Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · 7 citations) |
| Remove pretraining | halfcheetah-medium-v2 · normalized score | 57.9 -> 57.5 | No clear effect | Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · 6 citations) |
| Replace linear projections with MLP embeddings | offline RL tasks · normalized score | Helped | Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning (ICLR 2024 · 5 citations) | |
| Increase DT context length beyond 1 | D4RL · normalized return | No clear effect | When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · 5 citations) | |
| add policy re-evaluation and value alignment to O2SAC | hopper-expert-v2 online fine-tuning · normalized score | Helped | Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · 5 citations) | |
| Increase recursion depth from N=1 toward N=2 | D4RL benchmark tasks · normalized score | Helped | Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · 4 citations) | |
| remove offline constraints during online fine-tuning | Walker2d-medium · normalized return | Helped | SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning (AAAI 2024 · 4 citations) | |
| Add the multi-grained branch | Gym-MuJoCo (Avg.) · normalized score | 79.2 -> 83.2 | Helped | Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL (NeurIPS 2024 · 4 citations) |
| Use RNN-like full-trajectory input instead of Transformer-like fixed-length input | Atari, average · normalized score | 111.8 -> 67.3 | Hurt | Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning? (NeurIPS 2024 · 4 citations) |
| use HJ reachability with a feasible value function instead of cost value function to identify feasibility | CarRun · normalized return | -0.33 -> 0.73 | Helped | Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · 3 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| CDT | BulletGym average | normalized reward | 0.63 – 0.68 | 2 papers, e.g. ICML 2025 and NeurIPS 2025 |
What reviewers will expect
Datasets and baselines named in the 26 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- D4RL antmaze · 3 of 26 papers
- OGBench pointmaze · 3 of 26 papers
- OGBench antmaze · 3 of 26 papers
- OGBench humanoidmaze · 3 of 26 papers
- OGBench cube · 3 of 26 papers
- OGBench scene · 3 of 26 papers
- DSRL benchmark (Safety-Gymnasium, Bullet-Safety-Gym, MetaDrive) · 2 of 26 papers
- OGBench Puzzle · 2 of 26 papers
Baselines
- CPQ · 6 of 26 papers
- CDT · 6 of 26 papers
- COptiDICE · 6 of 26 papers
- CQL · 4 of 26 papers
- IQL · 4 of 26 papers
- HIQL · 4 of 26 papers
- BEAR-Lag · 4 of 26 papers
- BCQ-Lag · 4 of 26 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 83 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 16 papers
- Google · 8 papers
- University of California, Berkeley · 5 papers
- Carnegie Mellon University · 5 papers
- Massachusetts Institute of Technology · 4 papers
- Microsoft · 4 papers
- Meta · 4 papers
- Stanford University · 4 papers
- Zhejiang University · 4 papers
- Shanghai Jiao Tong University · 4 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Diffusion models for offline decision making (82 papers)
- Decision Transformer sequence models (40 papers)
- Offline policy evaluation and selection (8 papers)
Elsewhere in the topic
- Offline RL sample complexity theory (156 papers)
- Simple practical offline RL (144 papers)
- Pessimism and uncertainty in offline RL (59 papers)
- Behavior regularization for offline RL (35 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).