Research map / Reinforcement learning / Offline RL
ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Advantage-Conditioned Transformer (ACT), which labels offline trajectories with dynamic-programming-based advantage estimates (IAE or GAE) via in-sample expectile value iteration and conditions an encoder-decoder transformer on the desired advantage instead of return-to-go.
Problem
Decision Transformer conditions on return-to-go, which requires complete future trajectories, cannot stitch trajectory segments, and over-optimistically exploits high returns in stochastic environments.
Approach
Replace RTG conditioning with advantages estimated by in-sample expectile value iteration (IAE = Q - V, or GAE), and use an encoder-decoder transformer with a learned predictor of the maximal in-sample advantage as the test-time target.
Key results
- On deterministic D4RL MuJoCo tasks ACT achieves the highest score among sequence-modeling methods in 8 of 9 tasks (e.g., hopper-medium-replay 98.4 vs DT 75.8).
- On the 2048 stochastic game ACT converges to a 100% success rate while DT and 10%-BC converge to suboptimal policies.
- On delayed-reward tasks DT achieves average 88.0 vs ACT 81.7, with ACT outperforming CQL (13.4) and IQL (62.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RTG conditioning with advantage conditioning | Decision Transformer, yielding ACT · Deterministic Gym MuJoCo | normalized score | Helped |
| Use ACT instead of the strongest listed baseline | HalfCheetah-medium · HalfCheetah-medium | normalized score: 48.4 -> 49.1\pm 0.2 | Helped |
| Use ACT instead of the strongest listed baseline | Hopper-medium-replay · Hopper-medium-replay | normalized score: 97.5 -> 98.4\pm 2.4 | Helped |
| Use ACT instead of the strongest listed baseline | Walker2d-medium-replay · Walker2d-medium-replay | normalized score: 77.2 -> 56.1\pm10.9 | Hurt |
| Use ACT instead of DT | ACT · Delayed-reward Hopper tasks | normalized score: 65.7_{\pm1.4} -> 49.4_{\pm1.5}; \bm{106.6_{\pm3.7}} -> 95.4_{\pm13.9} | Hurt |
| Use ACT instead of DT | ACT · Delayed-reward Walker2d tasks | normalized score: 72.1_{\pm4.3} -> \bm{73.8_{\pm1.9}}; 107.4_{\pm0.3} -> \bm{108.1_{\pm0.2}} | Helped |
| Increase predictor expectile σ2 (ablation) | ACT · D4RL tasks | performance | Helped |
| Replace the learned predictor with a constant positive target advantage (ablation) | ACT · HC-medium and HP-medium | performance | Mixed |
| Replace ACT encoder-decoder with GPT-2 architecture (ablation) | ACT · Hopper D4RL tasks | performance | Hurt |
| Replace sinusoidal encoding with learnable positional embedding (ablation) | ACT · D4RL tasks | performance | No clear effect |
| Use encoder-decoder architecture while retaining RTG conditioning (ablation) | DT-encdec · D4RL tasks | performance | Hurt |
| Remove predictor cφ and set desired RTG as in DT (ablation) | DT variant with ACT architecture · D4RL tasks | performance | Hurt |
Surprising findings
- On delayed-reward tasks DT outperforms ACT on average, and CQL degrades sharply, suggesting the advantage approach is not uniformly superior under sparse rewards.
- On NeoRL, CQL significantly outperforms ACT, IQL, and DT, which the authors attribute to CQL's search along the Q-function for OOD actions.
Limitations
- On delayed-reward tasks ACT is not uniformly better than DT and is affected by reward sparsity.
- Performance drops on NeoRL relative to CQL; ACT does not generate OOD actions beyond the dataset.
- Test-time advantage conditioning lacks a guarantee of positive performance difference when value estimation error is large (e.g., sigma_1 = 0.7 drops in some tasks).
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d 2048 game dataset stochastic Gym MuJoCo datasets NeoRL
Benchmarks: D4RL Gym MuJoCo 2048 game stochastic Gym MuJoCo delayed reward D4RL tasks NeoRL
Metrics: normalized score success rate
Baselines: CQL IQL Onestep RL 10% BC RvS DT QDT EDT ESPER
Builds on: Decision Transformer Implicit Q-learning Generalized Advantage Estimation Q-learning with transformer-based conditioning (QDT) Conservative Q-learning
Closest papers by meaning
- Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.88)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.86)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.