Research map / Reinforcement learning / Offline RL

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline-to-online fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diffusion Subgoal Planning (DSP), which generates high-level subgoals with a flow-matching diffusion model using classifier-free guidance instead of value-based selection, paired with a HIQL-style value-trained low-level policy, for offline goal-conditioned RL.

Problem

Offline goal-conditioned RL in long-horizon tasks suffers from noisy goal-conditioned value estimates, which destabilize high-level subgoal selection in hierarchical methods such as HIQL.

Approach

Replace value-based high-level subgoal selection with a conditional/unconditional flow-matching subgoal generator whose classifier-free guidance difference is argued to implicitly perform relative-advantage-weighted subgoal extraction, while keeping a value-trained low-level AWR policy.

Key results

• DSP achieves a total of 1090.2 across the OGBench subset versus 920.4 for the best baseline OTA (Table of results).
• On antmaze-giant-navigate, DSP reaches 69.2±2.9% success vs 68.4±3.4% for OTA and 58.2±4.7% for HIQL.
• On humanoidmaze-medium-navigate, DSP reaches 89.6±2.9% vs 82.4±3.6% for OTA.

Surprising findings

• Increasing the CFG guidance scale beyond moderate values degrades validity, reachability and success (e.g. Ant-Giant success drops from 69.2% at omega=3 to 27.0% at omega=10).
• DSP does not beat OTA on pointmaze-giant (43.4 vs 69.4) and humanoidmaze-large/giant, where OTA is stronger.

Limitations

• Performance may degrade under limited data coverage or evaluation goals far from the behavior data; no hard subgoal-feasibility guarantee.
• Iterative diffusion sampling for subgoals adds inference cost.
• Evaluated only on state-based OGBench settings, not partially observable domains.

Experimental setup

Datasets: OGBench pointmaze OGBench antmaze OGBench humanoidmaze OGBench antsoccer OGBench cube OGBench scene OGBench puzzle D4RL antmaze

Benchmarks: OGBench

Metrics: average binary success rate (%) support distance state validity rate valid-and-reached rate

Baselines: GCBC CFGRL GCIVL OTA Pi-HIQL HIQL

Builds on: HIQL implicit Q-learning classifier-free guidance flow matching OGBench

Closest papers by meaning

• Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.92)
• Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.91)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Learning to Reach Goals via Diffusion (ICML 2024 · similarity 0.91)
• Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.90)