Research map / Reinforcement learning / Offline RL
Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adding a weighted TD3 actor loss (with a critic conditioned only on state) to Online Decision Transformer finetuning, arguing high out-of-distribution return-to-go hampers ODT and RL gradients provide local improvement directions.
Problem
Online finetuning of decision transformers, especially when pretrained on low-reward offline data, where the widely used Online Decision Transformer (ODT) fails to improve.
Approach
Add a weighted TD3 actor loss (with a state-only critic) to the decision transformer's supervised objective during offline pretraining and online finetuning, supplying value-gradient directions that ODT's return-conditioning lacks; supported by a performance-bound analysis linking the far-from-data RTG to failure.
Key results
- On MuJoCo, TD3+ODT reaches an average final reward of 88.38 versus 87.64 for TD3 and 55.14 for ODT (table reports 88.38 average final for TD3+ODT).
- On Adroit, TD3+ODT averages 87.84 final reward versus 61.14 for IQL and 38.73 for ODT.
- On Antmaze umaze and medium mazes, TD3+ODT averages 93.17 versus 80.26 for IQL and 3.98 for ODT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TD3 gradients to ODT | ODT · MuJoCo average | normalized average reward: 55.14(+18.58) -> 88.38(+46.59) | Helped |
| Add TD3 gradients to ODT | ODT · Adroit average | average reward: 38.73(+3.75) -> 87.84(+33.09) | Helped |
| Add TD3 gradients to ODT | ODT · Antmaze umaze and medium average | average reward: 3.98(+22.52) -> 93.17(+51.17) | Helped |
| Use TD3 gradients alone instead of combining them with ODT supervised learning (ablation) | Decision Transformer online finetuning · MuJoCo average | average reward: 88.38(+46.59) -> 87.64(+44.95) | Hurt |
| Replace TD3 gradients with DDPG gradients (ablation) | ODT · MuJoCo average | average reward: 88.38(+46.59) -> 22.87(-19.22) | Hurt |
| Increase the TD3 loss coefficient alpha (ablation) | TD3+ODT · MuJoCo and Adroit environments | normalized average reward | Helped |
| Set the TD3 loss coefficient alpha too large (ablation) | TD3+ODT · MuJoCo and Adroit environments | stability | Hurt |
| Use an evaluation context length that is too long or too short (ablation) | TD3+ODT · HalfCheetah-medium-replay-v2 | average reward | Hurt |
| Replace TD3 gradients with SAC gradients in the mixed objective (ablation) | ODT · D4RL benchmarks | average reward | Hurt |
| Use IQL gradients instead of TD3 gradients (ablation) | ODT · Antmaze large | average reward | Helped |
| Use IQL gradients instead of TD3 gradients (ablation) | ODT · Low-reward offline data | average reward | Hurt |
| Add an action regularizer toward the pretrained policy (ablation) | TD3+ODT · Online finetuning environments | average reward | Mixed |
Surprising findings
- PPO-finetuned decision transformers were unstable because importance ratios become very small under mismatched RTG and context length.
- IQL gradients fail to improve when pretrained on very low-reward data such as hopper-random, although TD3 gradients succeed.
Limitations
- Theoretical analysis relies on strong assumptions (near determinism, Beta prior, Laplace RTG distributions).
- Evaluated only on low-dimensional state-based environments; image-based tasks left for future work.
- Additional computational cost relative to ODT.
Experimental setup
Datasets: D4RL Adroit D4RL Antmaze D4RL MuJoCo
Benchmarks: D4RL Adroit D4RL Antmaze D4RL MuJoCo
Metrics: normalized average reward rliable median, IQM, optimality gap
Baselines: ODT PDT TD3+BC TD3 IQL DDPG+ODT SAC+ODT AWAC
Builds on: Online Decision Transformer TD3 TD3+BC Brandfonbrener et al. performance bound AWAC
Closest papers by meaning
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.90)
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.89)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.