Research map / Reinforcement learning / Visual and representation RL
Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCE, an episodic RL method that splits each trajectory into K time-pair segments and uses segment-wise likelihood ratios and advantages with a full-covariance ProDMP policy and trust region projection, to get step-based sample efficiency with episodic exploration.
Problem
Episodic RL explores smoothly in trajectory-parameter space but treats each trajectory as a single data point, ignoring step-level information, which makes it sample-inefficient compared to step-based RL.
Approach
Replace trajectory-level likelihood and advantage in the ERL objective with segment-wise counterparts computed on K randomly selected time pairs of a ProDMP trajectory distribution with full covariance, enforced via trust region projection layers.
Key results
- On Metaworld across 50 tasks with 20 seeds, TCE significantly outperforms baselines in IQM success rate and surpasses other algorithms after roughly 1e7 environment interactions.
- On Table Tennis with sparse rewards, BBRL-Cov leads with about 20% higher success than other methods, TCE follows, and TRPL reaches only about 15% among step-based methods.
- On Hopper Jump and Box Pushing, TCE shows quick learning and high consistency, with gSDE competitive but highly seed-sensitive in Hopper Jump.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use segment-wise likelihoods and advantages for policy updates | TCE · Metaworld | success rate | Helped |
| use segment-wise likelihoods and advantages for policy updates | TCE · Hopper Jump | max jump height | Helped |
| use segment-wise likelihoods and advantages for policy updates | TCE · Box Pushing, dense reward | success rate | Helped |
| use segment-wise likelihoods and advantages for policy updates | TCE · Box Pushing, sparse reward | success rate | Helped |
| use segment-wise likelihoods and advantages for policy updates | TCE · Table Tennis | success rate | Mixed |
| replace the factorized policy with a full covariance policy (ablation) | BBRL · Hopper Jump | max jump height | Helped |
| replace the factorized policy with a full covariance policy (ablation) | BBRL · Box Pushing | success rate | Helped |
| replace the factorized policy with a full covariance policy (ablation) | BBRL · Table Tennis | success rate: 20% higher | Helped |
| use pink noise for exploration | SAC · Metaworld | final success rate | Helped |
| use pink noise for exploration | SAC · Metaworld | sample efficiency | Hurt |
| use gSDE exploration | PPO · Hopper Jump | max jump height | Helped |
| use full covariance movement correlations (ablation) | BBRL · Table Tennis | success rate: 20% higher | Helped |
Surprising findings
- Full-covariance BBRL (BBRL-Cov) outperforms TCE in the sparse-reward table tennis task, and TCE underperforms there due to reward sparsity hurting value learning.
- Step-based methods like PPO and TRPL struggle on box pushing even with dense rewards.
Limitations
- Moderate results on tasks with particularly sparse rewards such as table tennis.
- Requires a low-level tracking controller, which may be unsuitable for tasks like locomotion.
- Open-loop trajectory control lacks adaptability to dynamic environments.
Experimental setup
Datasets: Metaworld OpenAI Gym Hopper Franka Emika Panda box pushing robot table tennis
Benchmarks: Metaworld (50 tasks) Hopper Jump Box Pushing (dense and sparse) Table Tennis
Metrics: success rate (IQM) max jump height environment interactions to success 95% stratified bootstrap CI
Baselines: PPO SAC TRPL gSDE PINK BBRL-Std BBRL-Cov
Builds on: Black Box Reinforcement Learning Probabilistic Dynamic Movement Primitives Trust Region Projection Layers Generalized State Dependent Exploration Generalized Advantage Estimation
Closest papers by meaning
- TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.88)
- e-COP : Episodic Constrained Optimization of Policies (NeurIPS 2024 · similarity 0.87)
- Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning (ICLR 2024 · similarity 0.87)
- DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.