Research map / Reinforcement learning / Offline RL
TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TOP-ERL, an off-policy episodic RL method that uses a Transformer critic to estimate values of action segments from movement-primitive trajectories, trained with N-step returns and ProDMP initial-condition enforcement.
Problem
Episodic RL methods that predict whole action trajectories are typically restricted to on-policy updates because standard TD-based critics cannot evaluate action sequences, which limits sample efficiency.
Approach
A decoder-only Transformer critic takes the segment's starting state and its action sequence as tokens and outputs state-action values for every prefix, trained with N-step return targets on fixed replay actions (so no importance sampling), enabling off-policy ERL with random segment lengths and ProDMP-based initial-condition matching.
Key results
- On Meta-World MT50, TOP-ERL reaches 98% aggregated success after 10M samples, while TCE needs 20M to reach the same rate and SAC plateaus at 85%.
- On dense-reward Box Pushing, TOP-ERL reaches 80% success after 10M samples versus 60% for TCE after 50M samples.
- Ablation shows fixed 25-step segments drop dense Box Pushing success from 80% to 35%, so random segment lengths are the most influential component.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use off-policy Transformer critic updates with N-step returns instead of on-policy ERL updates | TOP-ERL · Dense Box Pushing | success rate: 80% success after just 10 million samples; TCE: 60% success after 50 million samples | Helped |
| use off-policy Transformer critic updates with N-step returns instead of on-policy ERL updates | TOP-ERL · Sparse Box Pushing | success rate: 70% success rate with 14 million environment interactions; TCE and gSDE require 50 million samples to reach 60% success | Helped |
| use off-policy Transformer critic updates with N-step returns instead of on-policy ERL updates | TOP-ERL · Meta-World MT50 | success rate: 98% after 10 million samples; TCE required 20 million interactions for the same success rate | Helped |
| use fixed 25-step segments instead of randomly sampled segment lengths (ablation) | TOP-ERL · Dense Box Pushing | success rate: 80% to 35% | Hurt |
| use fixed 25-step segments instead of randomly sampled segment lengths (ablation) | TOP-ERL · Sparse Box Pushing | success rate: 70% to 20% | Hurt |
| add dropout rate 0.05 to Transformer layers (ablation) | TOP-ERL · Box Pushing, dense and sparse rewards | success rate | Hurt |
| add layer normalization (ablation) | TOP-ERL · Box Pushing, dense and sparse rewards | success rate | Helped |
| remove trust region constraints from policy updates (ablation) | TOP-ERL · Box Pushing, dense and sparse rewards | success rate | Hurt |
| remove initial-condition enforcement for action segments during value-function training (ablation) | TOP-ERL · Box Pushing, dense and sparse rewards | value estimation performance | Hurt |
| use a single V target network rather than two target networks (ablation) | TOP-ERL · Box Pushing, dense and sparse rewards | success rate | No clear effect |
| use a single V target network rather than two target networks (ablation) | TOP-ERL · Box Pushing | computation time per update: approximately 50% faster | Helped |
| use the task initial state for every segment instead of each segment's own initial state (ablation) | TOP-ERL policy updates · Box Pushing | policy performance | Helped |
Surprising findings
- Adding dropout at 0.05 hurt performance in both Box Pushing settings.
- Using the task initial state for all segments in the policy objective performed better than per-segment initial conditions, despite the latter being more consistent for the critic.
Limitations
- Generates trajectories only at episode start, so it cannot handle dynamic target changes or replanning within an episode.
- Not designed for POMDPs; the Transformer processes action sequences rather than state histories.
Experimental setup
Datasets: Meta-World MT50 HopperJump Box Pushing (dense and sparse reward)
Benchmarks: Meta-World MT50 HopperJump Box Pushing dense reward Box Pushing sparse reward
Metrics: success rate interquartile mean with 95% bootstrap CI sample efficiency computation time per update
Baselines: BBRL TCE PPO SAC gSDE PINK GTrXL
Builds on: SAC ProDMP BBRL TCE TRPL GTrXL
Closest papers by meaning
- Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning (ICLR 2024 · similarity 0.90)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.86)
- Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks (AAAI 2025 · similarity 0.86)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.86)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.