Research map / Reinforcement learning / Offline RL
Q-value Regularized Transformer for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-value regularized Transformer (QT), which adds a Q-value maximization term to Decision Transformer-style conditional sequence modeling and uses Q-guided action selection at inference, to improve stitching and long-horizon offline RL on D4RL.
Problem
Conditional sequence modeling (e.g., Decision Transformer) struggles to stitch optimal trajectories from sub-optimal data because sampled return-to-go values are inconsistent with optimal future returns, while Q-learning methods are unstable in long-horizon and sparse-reward settings.
Approach
Combine a Transformer conditional behavior-cloning loss with a Q-value maximization term learned via n-step double-Q targets, and at inference sample actions over multiple candidate return-to-go values and pick the one with highest Q-value.
Key results
- QT averages 89.8 on D4RL Gym tasks versus 82.4 for the best listed baseline (CGDT) and 81.8 for DD.
- On Maze2D, QT averages 172.5 versus 121.5 for DD and 57.2 for BEAR; on maze2d-large QT reaches 240.1.
- On walker2d-medium-replay ablation, the full QT reaches 98.5 versus 87.6 for DT with Q-inference only and 79.4 for the CTP baseline without Q-value.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add n-step Q-value update, training regularization, and Q-value inference to BC (ablation) | BC policy · walker2d-medium-replay-v2 | normalized score: 32.3 $\pm$ 9.8 -> 82.2 $\pm$ 0.5 | Helped |
| Add Q-value inference (ablation) | Conditional Transformer Policy · walker2d-medium-replay-v2 | normalized score: 79.4 $\pm$ 2.0 -> 87.6 $\pm$ 1.1 | Helped |
| Add Q-value regularization during training (ablation) | Conditional Transformer Policy · walker2d-medium-replay-v2 | normalized score: 79.4 $\pm$ 2.0 -> 97.7 $\pm$ 0.3 | Helped |
| Use n-step instead of 1-step Q-value updates (ablation) | QT · walker2d-medium-replay-v2 | normalized score: 85.6 $\pm$ 1.7 -> 98.5 $\pm$ 1.1 | Helped |
| Increase η from 0.01 to 2 (ablation) | QT · walker2d-medium-replay-v2 dense reward | normalized score: $88.0 \pm 0.4$ -> $98.5 \pm 1.1$ | Helped |
| Increase η from 0.1 to 1 (ablation) | QT · walker2d-medium-replay-v2 sparse reward | normalized score: $72.3 \pm 0.3$ -> $7.0 \pm 4.6$ | Hurt |
| Use QT rather than the strongest listed baseline | D4RL Gym methods · D4RL Gym average | normalized score: 81.8 -> 89.8 | Helped |
| Use QT rather than the strongest listed baseline | D4RL Adroit methods · D4RL Adroit average | normalized score: 38.9 -> 60.4 | Helped |
| Use QT rather than the strongest listed baseline | D4RL Kitchen methods · D4RL Kitchen average | normalized score: 72.3 -> 78.4 | Helped |
| Use QT rather than the strongest listed baseline | D4RL Maze2D methods · D4RL Maze2D average | normalized score: 121.5 -> 172.5 | Helped |
| Use QT rather than the strongest listed baseline | D4RL AntMaze methods · D4RL AntMaze average | normalized score: 73.7 -> 76.5 | Helped |
| Use QT rather than CQL | Offline RL methods · 1% DQN-replay Atari average | score: 107.2 -> 145.7 | Helped |
Surprising findings
- In sparse-reward settings (delayed rewards), the Q-value regularization weight eta must be small; performance drops sharply at eta=1 (7.0) versus 78.5 at eta=0.01 on walker2d-medium-replay sparse.
- QT's advantage over QDT and CQL on delayed-reward MuJoCo tasks shows CQL alone performs poorly (average 7.2) under sparse reward.
Limitations
- Requires explicit reward signals; may be limited on datasets with only state-action pairs such as human demonstrations.
- Sensitive to the Q-value regularization weight eta, which must be tuned per domain and is harmed when Q-value estimation is inaccurate (sparse rewards).
- Can underperform on dense maze2d reward settings where reward structure inflates values of failing trajectories.
Experimental setup
Datasets: D4RL Gym D4RL Adroit D4RL Kitchen D4RL Maze2D D4RL AntMaze Atari 1% DQN-replay
Benchmarks: D4RL Gym D4RL Adroit D4RL Kitchen D4RL Maze2D D4RL AntMaze Atari 1% DQN-replay
Metrics: normalized score average return
Baselines: CQL IQL BCQ BEAR TD3+BC MoRel BC Diffuser DD DT StAR GDT
Builds on: Decision Transformer Conservative Q-learning Q-learning with Decision Transformer (QDT) TD3+BC minGPT
Closest papers by meaning
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.89)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.89)
- ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning (AAAI 2024 · similarity 0.89)
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.