Research map / Reinforcement learning / Offline RL
Decision Transformer sequence models
Cast offline RL as sequence modeling with transformers, enhanced by value guidance.
40 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Offline RL, keyword cluster “offline online · transformer · online reinforcement”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 15 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.5% to 6.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Decision Transformer: Reinforcement Learning via Sequence Modeling (NeurIPS 2021 · 464 citations)
- 2022 · Prompting Decision Transformer for Few-Shot Policy Generalization (ICML 2022 · 18 citations)
- 2023 · Is Conditional Generative Modeling all you need for Decision Making? (ICLR 2023 · 31 citations)
- 2024 · Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · 19 citations)
- 2025 · Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · 2 citations)
- 2026 · DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation (ICML 2026 · 0 citations)
Earliest papers here
- Decision Transformer: Reinforcement Learning via Sequence Modeling (NeurIPS 2021 · 464 citations)
- Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (ICLR 2021 · 8 citations)
- Prompting Decision Transformer for Few-Shot Policy Generalization (ICML 2022 · 18 citations)
What has been tried
30 papers here report 318 measured changes (a component added, removed or swapped, compared with a baseline): helped 55%, no clear effect 12%, hurt 25%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Set critic-training asymmetry τc above 0.5 to bias fitting toward high-quality data | D4RL tasks (excluding Antmaze tendency) · normalized score | Helped | Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · 19 citations) | |
| Increase predictor expectile σ2 | D4RL tasks · performance | Helped | ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning (AAAI 2024 · 12 citations) | |
| Add n-step Q-value update, training regularization, and Q-value inference to BC | walker2d-medium-replay-v2 · normalized score | 32.3 ± 9.8 -> 82.2 ± 0.5 | Helped | Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · 4 citations) |
| Replace TSSA/PSSA with SDSA-1 | MuJoCo-Hopper · score | Hopper-m-e: 110.9 -> 104.3; Hopper-m: 74.1 -> 54.8; Hopper-m-r: 96.3 -> 47.1 | Hurt | Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · 2 citations) |
| Use TD3 gradients alone instead of combining them with ODT supervised learning | MuJoCo average · average reward | 88.38(+46.59) -> 87.64(+44.95) | Hurt | Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers (NeurIPS 2024 · 2 citations) |
| learn task-specific masks instead of keeping random masks fixed | Meta-World MT50-rand near-optimal · average success rate | 75.39 (± 1.18) -> 80.33 (± 0.97) (HarmoDT-M); 75.39 (± 1.18) -> 82.20 (± 0.40) (HarmoDT-F) | Helped | HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning (ICML 2024 · 1 citation) |
| Add block-wise masking rather than token-wise masking | Skill prompting (Average) · reward | 56.7 -> 35.0; 56.7 -> 73.4 | Mixed | Learning Versatile Skills with Curriculum Masking (NeurIPS 2024 · 1 citation) |
| Use longer context, k=10 to k=50 | PointMaze large · episode length | 236.3 -> 203.0 | Helped | Unveiling Markov heads in Pretrained Language Models for Offline Reinforcement Learning (ICML 2025 · 0 citations) |
| Use DIT’s exponential advantage-weighted pretraining instead of unweighted behavior cloning | Miniworld · episode cumulative return | Helped | In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · 0 citations) | |
| use fixed 25-step segments instead of randomly sampled segment lengths | Dense Box Pushing · success rate | 80% to 35% | Hurt | TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning (ICLR 2025 · 0 citations) |
| replace transformer dynamics model with MLP ensemble | PickPlace · success rate | Hurt | Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning (ICLR 2024 · 0 citations) | |
| remove the prefix robustness token | DSRL Run, Circle, and Reach · satisfaction rate | Hurt | Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning (ICML 2024 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| IQL | AntMaze average | normalized score | 57.3 – 63 | 2 papers, e.g. NeurIPS 2023 and NeurIPS 2023 |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- CQL · 3 of 11 papers
- DT · 3 of 11 papers
- BC · 2 of 11 papers
- SAC · 2 of 11 papers
- Behavior Cloning · 2 of 11 papers
- IQL · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 33 papers; from OpenAlex and the papers' affiliations).
- Shanghai Jiao Tong University · 7 papers
- Wuhan University · 3 papers
- University of California, Berkeley · 3 papers
- Tencent · 3 papers
- Nanjing University · 2 papers
- Shanghai AI Laboratory · 2 papers
- Covariant · 2 papers
- Tsinghua University · 2 papers
- Google · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Offline-to-online fine-tuning (99 papers)
- Diffusion models for offline decision making (82 papers)
- Offline policy evaluation and selection (8 papers)
Elsewhere in the topic
- Offline RL sample complexity theory (156 papers)
- Simple practical offline RL (144 papers)
- Pessimism and uncertainty in offline RL (59 papers)
- Behavior regularization for offline RL (35 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).