Research map / Reinforcement learning / Visual and representation RL
Learning to Reach Goals via Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Merlin reframes offline goal-conditioned RL as reversing a goal-anchored forward diffusion in state space, training a horizon-conditioned Gaussian policy by behavior cloning with one denoising step per environment step.
Problem
Offline goal-conditioned RL struggles with value function estimation from out-of-distribution actions and sparse binary goal rewards, and existing diffusion-based RL methods are computationally expensive because they denoise at every environment step.
Approach
Construct forward diffusion trajectories that move away from goal states (via reverse play, a learned reverse dynamics/policy model, or nearest-neighbor latent trajectory stitching), and train a goal- and horizon-conditioned Gaussian policy to reverse them via behavior cloning, with no value function and one denoising step per environment step.
Key results
- Merlin-P and Merlin-NP achieve the highest discounted returns on most tasks and are overall the best-performing methods across the offline goal-reaching benchmark.
- Merlin's training and inference are roughly an order of magnitude faster than diffusion-based baselines (DD, g-DQL, BESO).
- Conditioning on time horizon h is much better than no horizon conditioning on all tasks; HandReach is especially sensitive to h.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Merlin instead of the compared offline GCRL baselines | Offline goal-conditioned control methods · State-space tasks | discounted return | Helped |
| Add parametric reverse-trajectory generation | Merlin-P vs. Merlin · State-space tasks | discounted return | Helped |
| Add non-parametric trajectory stitching | Merlin-NP vs. Merlin · State-space tasks | discounted return | Helped |
| Use Merlin rather than diffusion-based RL methods | Merlin vs. DD, g-DQL, and BESO · State-space tasks | training and inference time | Helped |
| Condition the policy on the time horizon rather than omit horizon conditioning (ablation) | Merlin · Offline goal-conditioned control tasks | performance | Helped |
| Evaluate with time horizon h=1 rather than other values or no horizon conditioning (ablation) | Merlin on HandReach · HandReach | performance | Helped |
| Vary the hindsight relabeling ratio by dataset type: low-to-moderate for expert and high for random data (ablation) | Merlin · Expert and random offline datasets | performance | Mixed |
| Condition GCSL on the time horizon without learning policy variance (ablation) | GCSL · Offline goal-conditioned control tasks | performance | No clear effect |
| Use a backward-view trajectory construction rather than the corresponding forward-view variant | Merlin-P and Merlin-NP · Offline goal-conditioned control tasks | performance | Helped |
Surprising findings
- GCSL performs similarly with and without conditioning on the time horizon, whereas Merlin's learned variance and horizon conditioning matter substantially.
- The backward (reverse-play) view outperforms the forward-view variants of Merlin-P and Merlin-NP on most tasks.
Limitations
- Forward diffusion process construction is heuristic; three alternatives are explored and the best depends on the task.
- Nearest-neighbor stitching requires a carefully chosen cosine threshold (0.9999) and contrastive encoder, and may connect non-reachable states across barriers.
- Out-of-distribution goal generalization is weaker in regions poorly covered by forward diffusion data (e.g., far quadrant in four-rooms).
Experimental setup
Datasets: offline goal-conditioned benchmark (PointReach, PointRooms, Reacher, SawyerReach, SawyerDoor, FetchReach, FetchPush, Fet
Benchmarks: PointReach PointRooms Reacher SawyerReach SawyerDoor FetchReach FetchPush FetchPick FetchSlide HandReach
Metrics: discounted return success rate training time inference time
Baselines: GCSL WGCSL ActionableModel GoFAR Decision Diffuser g-DQL BESO
Builds on: GCSL Denoising diffusion probabilistic models Hindsight experience replay Decision Diffuser Diffusion-QL
Closest papers by meaning
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.90)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
- Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous and Instruction-guided Driving (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.