Research map / Reinforcement learning / Visual and representation RL
Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization for Goal-Conditioned Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes margin-based policy self-regularization (MSR), a KL-based plug-in loss that keeps policies for adjacent goals similar but not identical, improving continuity of goal achievement in GC-SAC, HER and GC-PPO.
Problem
Goal-conditioned RL policies show discontinuous goal-achievement: they often fail on goals just adjacent (within the success threshold) to goals they already achieve.
Approach
Theoretical bounds on return gaps between adjacent goals motivate a hinge-style regularizer that penalizes KL divergence between policies for a goal and its perturbed neighbors only above a margin beta, balancing reuse of successful policies against over-triviality.
Key results
- On Reach, MSR-GC-SAC raises adjacent-goal success rate to 99.98% versus 65.58% for GC-SAC, and overall success rate to 99.95% versus 0.20%.
- On VVC, MSR-GC-PPO improves overall success rate to 26.52% from 7.62% for GC-PPO (high variance).
- MSR adds about 2.14% wall-clock overhead for on-policy PPO and 16-21% for off-policy SAC/HER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSR to GC-SAC | GC-SAC · Reach | J(π): –12.03±0.63 -> -0.85±0.02 | Helped |
| Add MSR to HER | HER · Reach | J(π): -1.16±0.10 -> -0.96±0.08 | Helped |
| Add MSR to GC-PPO | GC-PPO · Reach | J(π): -24.89±1.62 -> -1.03±0.15 | Helped |
| Add MSR to GC-SAC | GC-SAC · Push | J(π): -25.50±8.00 -> -23.80±6.76 | Helped |
| Add MSR to HER | HER · Push | J(π): -18.00±2.24 -> -16.31±1.92 | Helped |
| Add MSR to GC-PPO | GC-PPO · Push | J(π): -24.76±2.99 -> -23.08±5.06 | Helped |
| Add MSR to GC-SAC | GC-SAC · VVC | J(π): -138.20±14.16 -> -132.09±8.06 | Helped |
| Add MSR to HER | HER · VVC | J(π): -69.21±12.87 -> -64.73±13.55 | Helped |
| Add MSR to GC-PPO | GC-PPO · VVC | J(π): -169.04±25.57 -> -146.64±28.05 | Helped |
| Increase the training perturbation range ϵ′ toward δ (ablation) | MSR-GC-SAC · Reach | DKL(πg+ϵ, πg) | Helped |
| Increase MSR strength λ (ablation) | MSR-GC-SAC · Reach | J(π) | Mixed |
| Increase MSR threshold β (ablation) | MSR-GC-SAC · Reach | DKL(πg+ϵ, πg) | Helped |
Surprising findings
- For GC-PPO on Reach and VVC, the policy KL between adjacent goals is small mainly because the policy barely achieves goals, so the KL metric is not reliable there.
- MSR was robust to the number of perturbation samples N (1 to 64).
Limitations
- Evaluated only on three tasks; scalability to more complex environments and other GCRL algorithms is untested.
- Does not address worst-case discontinuity of goal-achievement ability.
- Theoretical analysis of the link between policy differences and cumulative rewards could be tighter.
Experimental setup
Datasets: Panda-Gym Reach Panda-Gym Push FlyCraft VVC
Benchmarks: Reach (Panda-Gym) Push (Panda-Gym) VVC (FlyCraft)
Metrics: policy KL divergence between adjacent goals return gap J(pi_g+eps)-J(pi_g) expected return J(pi) adjacent success rate overall success rate
Baselines: GC-SAC HER GC-PPO
Builds on: GC-SAC HER GC-PPO TRPO Stable-Baselines3
Closest papers by meaning
- Goal-Conditioned On-Policy Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.88)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.88)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.87)
- Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.