Research map / Reinforcement learning / Offline RL
When should we prefer Decision Transformers for Offline Reinforcement Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically compares Conservative Q-Learning, Behavior Cloning, and Decision Transformer offline RL across data quality, trajectory length, random-noise, task complexity, and stochasticity on D4RL, robomimic, and Atari to give guidance on which algorithm to prefer.
Problem
Offline RL practitioners lack guidance on which algorithm (Q-learning, imitation learning, or sequence modeling) is preferable under different data, task, and environment conditions.
Approach
Targeted controlled experiments varying data suboptimality (best/worst X%, trajectory length, injected random data), task complexity (state dimension, horizon), and stochasticity, comparing CQL, BC, and DT under matched settings, plus DT architecture and scaling studies on Atari.
Key results
- On robomimic MG data, DT outperformed CQL by 154% and BC by 51% in dense reward; in sparse reward DT was 88% above CQL on D4RL.
- CQL was most sample-efficient on small amounts of high-quality data (~5% of dataset), while DT needed ~20%; DT never worsened with more data.
- On Atari, scaling data by 5x gave a 2.5x average score improvement for DT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DT instead of CQL | Offline RL · D4RL, sparse-reward tasks | normalized return: 40.78 ± 7.88 -> 76.6 ± 1 | Helped |
| Use CQL instead of DT | Offline RL · D4RL, dense-reward tasks | normalized return: 77.51 ± 1.12 -> 83.64 ± 0.51 | Helped |
| Redistribute each trajectory's reward to its last timestep | D4RL offline datasets · D4RL | normalized return: CQL's performance ... reduce by half; DT's performance remained unchanged | Mixed |
| Train on the best 5% rather than the best 20% of trajectories | CQL · D4RL dense-reward data | normalized return: CQL ... reaching its highest score at ~5% ... DT required ~20% | Helped |
| Add more, potentially lower-return trajectories | CQL · D4RL medium-replay | normalized return: between 20% and 80% | Hurt |
| Add more trajectories, including suboptimal data | DT · D4RL | normalized return | Helped |
| Use DT instead of CQL on low-quality medium-replay data | Offline RL · D4RL medium replay | learning speed: 33% faster on average | Helped |
| Use BC instead of DT | Offline RL · Robomimic human-generated data | success rate: 91.3 ± 0.9 vs. 83.1 ± 0.8 | Helped |
| Add random-action data to a high-quality human dataset | BC · Robomimic PH | success rate: nearly 100% deterioration on Lift PH | Hurt |
| Add an equal amount of random-action data | DT compared with BC · Offline RL noisy-data setting | evaluation score: less than 10% deterioration | Helped |
| Increase task horizon | BC and DT compared with CQL · Robomimic PH | success rate | Helped |
| Use CQL instead of DT or BC | Offline RL · D4RL medium-replay, high stochasticity (p=0.5, σ=0.5) | evaluation score | Helped |
Surprising findings
- CQL's performance drops by about half on D4RL when identical transitions are given sparse rewards, while DT is unchanged.
- BC outperforms DT and CQL on human-generated robomimic data (PH/MH), reversing the ranking seen on machine-generated data.
Limitations
- Only three algorithms (CQL, BC, DT) studied; IQL, Trajectory Transformer, model-based and diffusion methods excluded due to compute budget.
- Evaluated only on deterministic, fully observed tasks; stochasticity is simulated by injecting Gaussian action noise at evaluation.
- Sparse-reward D4RL construction may break Markovian dynamics.
Experimental setup
Datasets: D4RL robomimic Atari DQN replay dataset humanoid (generated by authors) Adroit Pen-human-v0 Maze2D Antmaze-umaze
Benchmarks: D4RL robomimic Atari Adroit Pen-human-v0 Maze2D Antmaze-umaze Humanoid
Metrics: normalized average return success rate expert-normalized score relative deterioration
Baselines: CQL BC DT
Builds on: Decision Transformer Conservative Q-Learning D4RL robomimic
Closest papers by meaning
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.90)
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.90)
- Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers (NeurIPS 2024 · similarity 0.90)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.89)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.