Research map / Reinforcement learning / Offline RL
Online Decision Transformer
Qinqing Zheng, Amy Zhang, Aditya Grover
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune offline-pretrained ODT with 200k online samples | ODT · D4RL Gym tasks and datasets | normalized return improvement from offline to online: 49.72 | Improved |
| finetune offline-pretrained IQL with 200k online samples | IQL · D4RL Gym tasks and datasets | normalized return improvement from offline to online: 5.56 | Improved |
| finetune offline-pretrained ODT with 200k online samples | ODT · AntMaze tasks | normalized return improvement from offline to online: 43.39 | Improved |
| finetune offline-pretrained IQL with 200k online samples | IQL · AntMaze tasks | normalized return improvement from offline to online: -5.2 | Hurt |
| use a stochastic policy rather than a deterministic DT-style policy (ablation) | ODT online finetuning · Hopper | return and training stability | Improved |
| use curriculum exploration RTGs rather than a fixed scaled RTG (ablation) | ODT online finetuning · Hopper | evaluation return | Hurt |
| set exploration RTG to twice rather than once the expert return (ablation) | ODT · Hopper | normalized return: 95.70 -> 97.54 | Improved |
| set exploration RTG to twice rather than once the expert return (ablation) | ODT · AntMaze umaze-diverse | normalized return: 59.4 -> 58.19 | Hurt |
| add hindsight return relabeling to correct rollout RTG tokens (ablation) | ODT online finetuning · Hopper medium | return | Improved |
| initialize replay buffer with top-return offline trajectories rather than random trajectories (ablation) | ODT · multiple environments | return | Improved |
| use evaluation context length 1 rather than 5 (ablation) | ODT online finetuning · Ant medium-replay | return | Improved |
| remove positional embeddings (ablation) | ODT online finetuning · Walker medium | return | Improved |
Closest papers by meaning
- Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers (NeurIPS 2024 · similarity 0.93)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.89)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.88)
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.88)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
