Research map / Probabilistic and scientific modeling / Molecules and proteins

Goal-Space Planning with Subgoal Models

ICML 2025 · Probabilistic and scientific modeling › Molecules and proteins · approach: Probabilistic embeddings and relational inference

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Goal-Space Planning (GSP), which runs background value iteration over a small set of subgoals using learned local subgoal-conditioned reward and discount models, then projects subgoal values onto states via potential-based reward shaping added to Sarsa(λ) and DDQN.

Problem

Background planning with learned one-step or multi-step models in model-based RL is often worse than model-free methods due to model inaccuracy, invalid generated states, compounding error, and high computational cost.

Approach

Constrain background planning to a finite set of abstract subgoals and learn only local, subgoal-conditioned reward and discount models (UVFA-style, outputting scalars), avoiding next-state prediction; subgoal values are projected back to states via potential-based shaping, preserving the optimal policy.

Key results

• GSP-augmented Sarsa(λ) in FourRooms reaches the optimal policy much faster than Sarsa(λ) with a lower initial steps-to-goal, averaged over 100 runs.
• In GridBall and PinBall with linear tile-coded learners, GSP significantly speeds up learning, with the base learner reaching near top performance within 75 and 100 episodes in GridBall and PinBall respectively.
• Approximate LAVI, which bootstraps directly off subgoal values, either learns much slower or converges to a suboptimal policy in all environments.

Measured effects reported in the paper

Surprising findings

• Using GSP out of the box with DDQN in PinBall converged to a suboptimal policy because shaping terms were sometimes an order of magnitude larger than environment reward; clipping or scaling the potential difference fixed this.
• Subgoal placement that biases the initial path toward a suboptimal alternative can cause GSP to learn a suboptimal policy before any environment change.

Limitations

• Assumes subgoals and initiation sets are given rather than discovered; subgoal selection affects performance and poor subgoals can harm learning.
• Potential-based shaping with large potential differences can destabilize deep learners (DDQN) without magnitude control.
• Only TD learners are studied; policy-gradient methods gain little from potential-based shaping and are left to future work.

Experimental setup

Datasets: FourRooms PinBall GridBall

Benchmarks: FourRooms GridBall PinBall

Metrics: steps to goal return action-value and state-value change after one episode probability of taking alternative path

Baselines: Sarsa(λ) Sarsa(0) DDQN Approximate LAVI Dyna with options Clipped GSP L2 distance potential random potential

Builds on: Dyna options LAVI Double DQN potential-based reward shaping

Closest papers by meaning

• Probabilistic Hierarchical Goal Network Planning with UCT (AAAI 2026 · similarity 0.84)
• Satisficing and Optimal Generalised Planning via Goal Regression (AAAI 2026 · similarity 0.84)
• Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.82)
• Efficient Robotic Policy Learning via Latent Space Backward Planning (ICML 2025 · similarity 0.82)
• Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.82)
• PcLast: Discovering Plannable Continuous Latent States (ICML 2024 · similarity 0.82)