Research map / Reinforcement learning / Offline RL
Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSD, a model-based offline goal-conditioned RL method that trains a goal-conditioned critic with multi-step goal chaining and a value-conditioned Unet-transformer diffusion model that generates sub-trajectories stitched toward the goal.
Problem
Offline goal-conditioned RL with sparse rewards from suboptimal data and long horizons, where existing diffusion planners (Diffuser, Decision Diffuser) are hard to apply because of inpainting with known plan length or unrealistic trajectory generation.
Approach
Condition a diffusion model on goal and a target value estimated by a goal-chaining critic trained on goal-relabeled sub-trajectories, using a Condition-Prompted-Unet with transformer blocks so the planner generates realistic sub-plans that stitch dataset segments without needing the plan length or explicit subgoals.
Key results
- On Maze2D single-task average, SSD scores 154.2 versus 130.5 for the best Diffuser variant (Diffuser-HER) and 47.0 for IQL.
- On Multi2D multi-task average, SSD scores 168.8 versus 139.6 for Diffuser-AM and 16.9 for IQL.
- On Fetch, SSD achieves the highest average success rate (0.74) versus 0.73 for GoFAR and 0.60 for GCSL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSD instead of the strongest listed baseline | Maze2D · Maze2D Umaze, Medium, and Large | normalized score: 144.6 ±3.4 vs 125.4 ±0.4; 134.4 ±6.1 vs 130.3 ±2.3; 183.5 ±8.6 vs 135.8 ±8.5 | Helped |
| Use SSD instead of the strongest listed baseline | Multi2D · Multi2D Umaze, Medium, and Large | normalized score: 158.2 ±4.5 vs 135.4 ±2.0; 155.2 ±7.9 vs 137.8 ±1.0; 192.9 ±8.5 vs 145.7 ±7.7 | Helped |
| Use SSD instead of the strongest listed baseline | Fetch · FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide | success rate: 1.0 ±0.0 vs 1.0 ±0.0; 0.9 ±0.03 vs 0.84 ±0.12; 0.97 ±0.02 vs 0.88 ±0.09; 0.1 ±0.07 vs 0.18 ±0.12 | Mixed |
| Use SSD instead of the strongest listed baseline | Fetch · FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide | discounted return: 29.18 ±0.32 vs 30.1 ±0.32; 18.0 ±0.25 vs 19.7 ±2.57; 19.84 ±0.50 vs 18.2 ±3.00; 2.6 ± 0.94 vs 2.73 ±1.64 | Mixed |
| Change the target value across tested values (ablation) | SSD · Maze2D Umaze | normalized score: 142.1 ±5.8 (v=0.0125); 134.7 ±6.9 (v=0.025); 142.1 ±3.4 (v=0.05); 149.1 ±5.4 (v=0.1); 144.6 ±3.4 (v=0.2); 150.3 ±7.4 (v= | Mixed |
| Change the target value across tested values (ablation) | SSD · Maze2D Medium | normalized score: 131.4 ±5.4 (v=0.0125); 137.3 ±5.5 (v=0.025); 134.4 ±6.1 (v=0.05); 134.5 ±3.2 (v=0.1); 129.5 ±5.0 (v=0.2); 128.2 ±4.6 (v= | Mixed |
| Change the target value across tested values (ablation) | SSD · Maze2D Large | normalized score: 171.4 ±7.7 (v=0.0125); 170.6 ±4.2 (v=0.025); 184.9 ±9.8 (v=0.05); 183.5 ±8.6 (v=0.1); 175.2 ±12.7 (v=0.2); 161.3 ±8.8 (v | Mixed |
| Change target value from v=0.2 to v=0.4 (ablation) | SSD · Maze2D Umaze | normalized score: 144.6 ±3.4 -> 150.3 ±7.4 | No clear effect |
| Use AM rather than HER goal relabeling for Diffuser (ablation) | Diffuser · Multi2D tasks | normalized score: 128.9 ±1.8 -> 135.4 ±2.0; 127.2 ±3.4 -> 137.8 ±1.0; 132.1 ±5.8 -> 145.7 ±7.7 | Helped |
| Use HER rather than AM goal relabeling for Diffuser (ablation) | Diffuser · Maze2D Umaze and Medium | normalized score: 121.4 ±1.8 -> 125.4 ±0.4; 127.2 ±3.1 -> 130.3 ±2.3 | Helped |
Surprising findings
- Reproduced Diffuser-HER and Diffuser-AM variants outperform DD and HDMI on Maze2D and Multi2D, indicating that classifier-guided Diffuser variants are strong baselines.
- SSD has the worst average final distance among methods on Fetch (0.124, tied with DDPG, vs 0.052 for GoFAR), because it does not reduce distance once the goal is reached.
Limitations
- Fetch final distance is worse than baselines because the policy does not minimize distance after reaching the goal.
- Goal chaining is attempted only at time-step t+1 to limit overestimation, a heuristic choice.
- Evaluation target value v must be tuned per environment and task complexity.
Experimental setup
Datasets: D4RL Maze2D (Umaze, Medium, Large) Multi2D Fetch offline datasets (random and expert, from prior work)
Benchmarks: Maze2D Multi2D Fetch (FetchReach, FetchPickAndPlace, FetchPush, FetchSlide)
Metrics: normalized score success rate discounted return final distance
Baselines: IQL Diffuser Diffuser-HER Diffuser-AM Decision Diffuser HDMI GCSL WGCSL GoFAR AM DDPG
Builds on: Diffuser Actionable Models (AM) Hindsight Experience Replay Decision Diffuser DDPM
Closest papers by meaning
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
- OGBench: Benchmarking Offline Goal-Conditioned RL (ICLR 2025 · similarity 0.91)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.91)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.91)
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.91)
- Score Models for Offline Goal-Conditioned Reinforcement Learning (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.