Research map / Reinforcement learning / Offline RL
PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlanDQ, a two-level offline RL method pairing a diffusion sub-goal planner (D-Conductor) with a goal-conditioned diffusion Q-learning low-level policy (Q-Performer), evaluated on D4RL Gym, AntMaze, Kitchen and Calvin.
Problem
Offline RL has no unified algorithm that performs well both on long-horizon sparse-reward tasks (where diffusion sequence planners excel) and short-horizon dense-reward tasks (where value-learning methods excel).
Approach
Combine a Hierarchical Diffuser-style sub-goal planner at the high level with a goal-conditioned Diffusion-QL low-level policy trained with TD learning and intrinsic plus external rewards, so the low level handles short dense sub-tasks with Bellman-based value learning.
Key results
- PlanDQ averages 83.0 on AntMaze (Medium/Large/Ultra) vs 76.0 for HIQL, the strongest hierarchical baseline in that table.
- On D4RL Medium-Replay Gym tasks PlanDQ averages 81.0 vs 72.3 for HD, a reported gain of about 12%.
- PlanDQ reaches 73.4 Kitchen average and 45.0 on Calvin, the best among hierarchical methods reported in the tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DQL rather than Diffuser as the low-level policy | PlanDQ low-level policy · OpenMaze | RL performance: 74.9 ± 1.2 -> 90.4 ± 1.9 | Helped |
| use a Q-learning conductor at the high level (ablation) | PlanDQ hierarchical architecture · Gym-MuJoCo dense-reward tasks | normalized score | Hurt |
| replace Q-Performer with behavior cloning (ablation) | PlanDQ · Gym-MuJoCo Med-Replay | normalized score: 37.6 ± 0.4 vs 46.7 ± 0.1; 23.6 ± 3.1 vs 101.4 ± 0.6; 18.8 ± 2.0 vs 94.0 ± 0.2 | Hurt |
| replace Q-Performer with behavior cloning (ablation) | PlanDQ · AntMaze Ultra | success score: 40.0 ± 15.4 -> 66.7 ± 12.2 | Hurt |
| replace Q-Performer with behavior cloning (ablation) | PlanDQ · Kitchen Mixed and Partial | success score: 41.7 ± 5.1 -> 71.7 ± 2.7; 28.3 ± 8.8 -> 75.0 ± 7.1 | Hurt |
| set the sub-goal interval to K=8 instead of K=4 (ablation) | PlanDQ · Gym-MuJoCo tasks | average normalized score: 88.2 -> 84.7 | Hurt |
| set the AntMaze sub-goal interval to K=45 instead of K=30 (ablation) | PlanDQ · AntMaze Large | success score: 86.7 ± 8.7 -> 60.0 ± 12.6 | Hurt |
| use K=45 rather than K=30 (ablation) | PlanDQ · AntMaze Ultra | success score: 66.7 ± 12.2 -> 66.7 ± 12.2 | No clear effect |
| combine external and intrinsic rewards instead of using intrinsic rewards only (ablation) | PlanDQ Q-Performer · Gym-MuJoCo Med-Replay | performance | Helped |
| combine external and intrinsic rewards instead of using external rewards only (ablation) | PlanDQ Q-Performer · Gym-MuJoCo Medium | performance | Helped |
| use PlanDQ rather than HD | hierarchical offline RL · Gym-MuJoCo Med-Replay | normalized score: 12% | Helped |
| use PlanDQ rather than the strongest flat baseline | offline RL · Gym-MuJoCo short-horizon tasks | normalized score | Mixed |
Surprising findings
- Diffuser's learned value predictor converged to a local optimum in a toy OpenMaze task, scoring 74.9 vs 90.4 for DQL, attributed to regression without Bellman updates.
- Using Q-learning at the high level (PlanQQ, PlanQD) degraded performance on dense-reward Gym tasks, which the authors attribute to off-policy value estimation mismatch.
Limitations
- Diffusion-based high-level planner makes policy generation slower than value-based methods.
- High-level plan quality depends heavily on data quality, which hurts on noisy data.
- Uses a fixed sub-goal interval K rather than adaptive intervals.
Experimental setup
Datasets: D4RL Gym-MuJoCo (halfcheetah, hopper, walker2d medium, medium-replay, medium-expert) D4RL AntMaze medium/large AntMaze-Ultra D4RL Kitchen mixed/partial Calvin
Benchmarks: D4RL Gym-MuJoCo D4RL AntMaze AntMaze-Ultra D4RL Kitchen Calvin OpenMaze (toy)
Metrics: normalized return success rate
Baselines: CQL IQL Diffuser DD TT DQL HIQL HDMI DTAMP HD
Builds on: Diffuser Hierarchical Diffuser Diffusion-QL TD3+BC Director
Closest papers by meaning
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.86)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.85)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.85)
- Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.