Research map / Reinforcement learning / Visual and representation RL

Goal-Conditioned On-Policy Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Graph search and compositional goals · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GCPO, an on-policy goal-conditioned RL framework that pre-trains a policy from demonstrations via behavioral cloning and then runs KL-regularized online RL with a self-curriculum over goals, targeting both Markovian and non-Markovian reward tasks on fixed-wing UAV velocity vector control.

Problem

Hindsight Experience Replay (HER)-based goal-conditioned RL cannot densify rewards when the reward is non-Markovian (depends on multiple states/actions), so off-policy replay-based curricula fail on such multi-goal long-horizon tasks.

Approach

Replace HER with an on-policy framework: behavioral cloning pre-training supplies an initial goal-achieving capability for informative rewards, and an online self-curriculum estimates achieved-goal distribution (GMM over weighted historical successes) and samples harder goals via MEGA-style selection, with KL-regularized policy optimization toward the pre-trained policy.

Key results

• On the NMR VVC task, GCPO reaches 45.87% success with D0_E versus 0.04% for SAC+HER+MEGA... with the full demonstration set D3_E GCPO achieves 59.90% vs 42.77% for SAC+HER+MEGA... (SAC+HER+MEGA is about 20% of GCPO's performance as the paper states)
• On Reach with NMR, GCPO achieves 80.26% success vs 0.72% for SAC+HER+MEGA; on PointMaze NMR, 47.50% vs 4.17%.
• Online self-curriculum improves success by an average 8.2% over GCPO without self-curriculum; pre-training is necessary, since GCPO w/o pre-training achieves only 31.28% on D3_E-level settings.

Measured effects reported in the paper

Surprising findings

• Stacking consecutive states as policy input to make the NMR Markovian degraded performance for SAC+HER+MEGA, BC and GCPO alike.
• Demonstration quality (shorter trajectories) improved the BC policy but the GCPO policy trained on smoother, longer demonstrations performed better after self-curriculum learning.

Limitations

• Relies on simple components (behavioral cloning and GMM); alternatives not explored.
• Under sparse rewards, GCPO requires the pre-trained policy to achieve some goals; otherwise self-curriculum is ineffective.
• Does not explicitly incorporate specialized NMR methods; the simple frame-stacking extension degraded performance.

Experimental setup

Datasets: Fixed-wing UAV VVC demonstrations generated by PID controller Minari pointmaze-large-v1 demonstrations PandaReach demonstrations generated with PID reference

Benchmarks: Fixed-wing UAV velocity vector control (VVC) Reach (PandaReach-v3, modified) PointMaze (modified)

Metrics: success rate (%)

Baselines: SAC+HER+MEGA BC GCPO w/o pre-training GCPO w/o self-curriculum SAC+HER PPO RIG DISCERN

Builds on: Hindsight Experience Replay MEGA KL-regularized RL IRPO

Closest papers by meaning

• Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization for Goal-Conditioned Reinforcement Learning (ICML 2025 · similarity 0.91)
• Goal-conditioned Imitation Learning (NeurIPS 2019 · similarity 0.90)
• Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.89)
• Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.88)
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.87)