Research map / Reinforcement learning / Offline RL
Critic-Guided Decision Transformer for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Critic-Guided Decision Transformer (CGDT), which adds a learned Gaussian critic with asymmetric (expectile) losses to Decision Transformer so that actions' expected returns match target returns, improving RCSL in stochastic bandits and D4RL stitching tasks.
Problem
Return-conditioned supervised learning (RCSL) methods like Decision Transformer ignore stochastic transitions and the distribution of future returns, so sampled trajectory returns are inconsistent with expected action returns, hurting performance in stochastic environments and stitching from subopti
Approach
Train a Gaussian critic on offline data with an asymmetric NLL loss to estimate expected returns of actions, then guide the DT policy with an expectile-regression term that pushes predicted actions' expected returns toward the target return, combined with the standard RCSL action loss.
Key results
- CGDT achieves 741.5 total normalized score on MuJoCo medium/medium-replay/medium-expert vs 699.2 for DT and 722.5 for TT (Table 1).
- On Hopper-Medium CGDT scores 96.9 vs 67.5 for DT.
- On Antmaze Umaze-Diverse CGDT scores 71.0 vs 55.0 for DT; Antmaze sum 142.0 vs 116.0 for DT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add critic guidance to DT | DT / CGDT · D4RL medium and medium-replay locomotion tasks | normalized score | Helped |
| Add critic guidance to DT | DT / CGDT · D4RL medium-expert locomotion tasks | normalized score | Mixed |
| Add critic guidance to DT | DT / CGDT · D4RL Antmaze-Umaze | normalized score: 61.0 -> 71.0 | Helped |
| Add critic guidance to DT | DT / CGDT · D4RL Antmaze-Umaze-Diverse | normalized score: 55.0 -> 71.0 | Helped |
| Add critic guidance to DT | DT / CGDT · Bernoulli Bandit | average normalized score | Helped |
| Add critic guidance to DT | DT / CGDT · D4RL sparse-reward locomotion tasks | normalized score: 687.4 -> 710.5; 699.2 -> 741.5 | Mixed |
| Set critic-training asymmetry τc above 0.5 to bias fitting toward high-quality data (ablation) | CGDT · D4RL tasks (excluding Antmaze tendency) | normalized score | Helped |
| Use larger τp to favor optimistic actions (ablation) | CGDT · D4RL tasks (excluding Halfcheetah tendency) | normalized score | Helped |
| Use larger τc to bias critic fitting toward high-quality data (ablation) | CGDT · Antmaze | normalized score | Mixed |
| Use larger τp to prefer optimistic actions (ablation) | CGDT · Halfcheetah | normalized score | Hurt |
| Train the critic on 90% of highest-return trajectories rather than all trajectories (ablation) | CGDT · Hopper and Walker2d datasets | overall improvement over DT | Helped |
| Adjust guidance-term scaling and balance weight α appropriately (ablation) | CGDT · D4RL tasks | normalized score | No clear effect |
Surprising findings
- Halfcheetah critic overestimates out-of-distribution actions, so a smaller tau_p is preferred there, reversing the general trend.
- CGDT's gains over DT are larger with dense rewards than with sparse (delayed) rewards, even though hyperparameters were not re-tuned for sparse rewards.
Limitations
- Critic may overestimate out-of-distribution actions, requiring early stopping and validation-based monitoring.
- Hyperparameters (tau_c, tau_p, alpha) require per-domain tuning; not all combinations were tuned.
- Baseline results for many methods are taken from original papers rather than re-run under identical settings.
Experimental setup
Datasets: D4RL Halfcheetah D4RL Hopper D4RL Walker2d D4RL Antmaze Bernoulli bandit offline dataset
Benchmarks: D4RL MuJoCo locomotion (medium, medium-replay, medium-expert) D4RL Antmaze (umaze, umaze-diverse) Bernoulli bandit
Metrics: average normalized score distance to Bayes-optimal in bandit
Baselines: BC RvS DT QDT CQL IQL TT DoC ESPER
Builds on: Decision Transformer Online Decision Transformer Implicit Q-Learning RvS
Closest papers by meaning
- Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers (NeurIPS 2024 · similarity 0.91)
- Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.90)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.89)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.