Research map / Reinforcement learning / Visual and representation RL
Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View.
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes stitching in offline goal-/return-conditioned RL as combinatorial generalization, shows supervised OCBC methods (DT, RvS) fail to stitch, and proposes a temporal data augmentation (clustering + future-goal resampling) that improves stitching across 10 state- and image-based environments.
Problem
Whether supervised-learning-based outcome-conditioned behavioral cloning (OCBC) methods for RL can stitch together pieces of experience to solve (state, goal) pairs never seen together in training, a property associated with dynamic programming / TD learning.
Approach
Casts stitching as combinatorial generalization over (state, goal) pairs unseen together during training, shows OCBC training distributions differ from the BC-policy test distribution, and proposes temporal data augmentation that replaces goals with future states of trajectories passing through nearby (k-means cluster) waypoints.
Key results
- Temporal augmentation improves RvS on 5 of 6 state-based maze tasks, and improves RvS and DT on 4/4 and 3/4 image-based tasks respectively.
- Adding augmentation improves combinatorial generalization of SL-based approaches by up to a factor of 2.5 across 10 environments.
- Increasing DT dataset size to 10 million transitions or number of layers does not substantially improve DT's stitching success.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal data augmentation to OCBC methods | RvS and DT · 10 state- and image-based environments | generalisation improvement: up to a factor of $2.5$ | Helped |
| Add temporal data augmentation | RvS · State-based maze stitching tasks | goal-reaching success rate | Helped |
| Add temporal data augmentation | RvS and DT · Image-based goal- and return-conditioned tasks | combinatorial generalisation | Helped |
| Train DT on 10 million transitions instead of the smaller dataset (ablation) | DT · Point maze tasks | combinatorial generalisation: $10$ million | No clear effect |
| Increase the number of transformer layers (ablation) | DT · Point maze tasks | stitching capability | No clear effect |
| Vary the number of K-means centroids for temporal data augmentation (ablation) | Temporal data augmentation · Point maze-medium and ant maze-medium | performance | Helped |
| Remove the success-associated spurious relation by making data-collecting policies stochastic at states 0 and 1 (ablation) | DT · Stochastic-v1, navigate from state 1 to state 4 | successful seeds: $\nicefrac{3}{10}$; $10$ seeds | Hurt |
| Remove the success-associated spurious relation by making data-collecting policies stochastic at states 0 and 1 (ablation) | DT · Stochastic-v2, navigate from state 0 to state 4 | successful seeds: $10/10$; $0/10$ | Hurt |
Surprising findings
- Scaling DT data by 10x or increasing model depth does not fix its failure to stitch.
- Most test state-goal pairs in original D4RL maze datasets already appear in the training distribution, so good D4RL scores do not necessarily require stitching.
Limitations
- Augmentation requires a local distance metric in state space, which is hard to obtain in high-dimensional settings.
- Temporal augmentation is not guaranteed to succeed on high-dimensional image tasks.
- Two extra hyperparameters (K and epsilon) were not tuned.
Experimental setup
Datasets: D4RL point maze D4RL ant maze Miniworld collect (new)
Benchmarks: stitching point maze datasets stitching ant maze datasets image-based point maze Miniworld collect
Metrics: success rate
Baselines: Decision Transformer RvS
Builds on: Decision Transformer RvS D4RL GCSL
Closest papers by meaning
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.88)
- Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.88)
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.88)
- Horizon Generalization in Reinforcement Learning (ICLR 2025 · similarity 0.87)
- State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.