Research map / Reinforcement learning / Offline RL
Future-conditioned Unsupervised Pretraining for Decision Transformer
Zhihui Xie, Zichuan Lin, Deheng Ye, QIANG FU, Yang Wei, Shuai Li
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use future trajectory embeddings to condition action prediction during pretraining rather than mask the future embedding (ablation) | PDT · Gym MuJoCo medium and medium-replay | downstream normalized return | Improved |
| Add return-prediction-guided sampling of future embeddings during finetuning (ablation) | PDT · Gym MuJoCo, especially medium-replay | normalized return | Improved |
| Continue updating the policy during finetuning instead of freezing all parameters except the return predictor (ablation) | PDT · hopper-medium and hopper-medium-replay | downstream performance | Improved |
| Increase future-embedding regularization strength (ablation) | PDT · medium datasets | finetuning normalized return | Mixed |
| Increase future-embedding regularization to beta = 1 (ablation) | PDT · behavior diversity | behavior diversity | Hurt |
| Condition on different predicted-return percentiles rather than always selecting the highest-return future (ablation) | PDT · Gym MuJoCo after online exploration | normalized return | Mixed |
| Use reward-free future conditioning instead of reward-conditioned ODT pretraining | PDT versus ODT · Gym MuJoCo medium-replay datasets | finetuning normalized return | Improved |
| Use reward-free future conditioning instead of reward-conditioned ODT pretraining | PDT versus ODT · halfcheetah-jump | normalized return: -31.00 ± 49.08 -> 70.39 ± 16.56 | Improved |
| Use reward-free future conditioning instead of reward-conditioned ODT pretraining | PDT versus ODT · walker2d-jump | normalized return: 15.81 ± 14.75 -> 68.70 ± 2.90 | Improved |
| Use reward-free future conditioning instead of reward-conditioned ODT pretraining | PDT versus ODT · Generalization tasks sum | normalized return: 101.45 -> 268.21 | Improved |
| Use reward-free future conditioning instead of reward-conditioned ODT pretraining | PDT versus ODT · halfcheetah-forward-jump | normalized return: 87.27 ± 14.41 -> 83.80 ± 2.28 | Hurt |
| Use pretrained future-conditioned policy rather than training SAC from scratch | PDT versus SAC · Gym MuJoCo tasks | sample efficiency | Improved |
Closest papers by meaning
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers (NeurIPS 2024 · similarity 0.90)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.88)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
