Research map / Reinforcement learning / Visual and representation RL
PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
PIPER learns a high-level preference-based reward model from primitive-generated sparse-reward preferences with hindsight relabeling and a value-based primitive regularizer, relabeling the higher-level replay buffer to mitigate HRL non-stationarity on sparse robotic tasks.
Problem
Off-policy hierarchical RL with goal-conditioned feudal policies suffers from lower-level reward non-stationarity in the higher-level replay buffer and from infeasible subgoal generation, while sparse rewards make learning hard.
Approach
Replace human preference labels with primitive-in-the-loop preferences derived from environment sparse rewards (with hindsight goal relabeling), learn a higher-level reward model from these preferences, relabel the higher-level buffer with it so the reward is independent of changing lower-level behavior, and regularize subgoal selection with the lower-level value function.
Key results
- PIPER achieves greater than 50% success rates in challenging sparse-reward robotic environments where most baselines fail to make significant progress.
- RAPS outperformed PIPER only in the maze navigation environment, attributed to its hand-designed lower-level primitive.
- SAC-Dense outperforms PIPER on easier tasks (maze, push) but PIPER outperforms SAC-Dense on harder tasks such as pick and place and kitchen, where SAC-Dense fails.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PIPER instead of vanilla hierarchical SAC (HIER) | Hierarchical SAC · Maze navigation, pick and place, push, hollow, and kitchen | success rate | Helped |
| Use PIPER instead of Hindsight Actor Critic (HAC) | Hierarchical reinforcement learning · Maze navigation, pick and place, push, hollow, and kitchen | success rate | Helped |
| Use hierarchical PIPER instead of flat preference-based RFLAT | Preference-based reinforcement learning · Sparse-reward tasks | success rate | Helped |
| Use PIPER instead of RAPS with hand-designed primitives | Hierarchical reinforcement learning · Maze navigation | success rate | Hurt |
| Use PIPER instead of RAPS with hand-designed primitives | Hierarchical reinforcement learning · Pick and place, push, hollow, and kitchen | success rate | Helped |
| Use PIPER instead of SAC-Dense | Sparse-reward reinforcement learning · Maze navigation and push | success rate | Hurt |
| Use PIPER instead of SAC-Dense | Sparse-reward reinforcement learning · Pick and place and kitchen | success rate | Helped |
| Add primitive-informed value regularization (ablation) | PIPER · Pick and place and push | success rate | Helped |
| Add primitive-informed value regularization (ablation) | PIPER · Maze, hollow, and kitchen | success rate | Helped |
| Vary the primitive regularization weight α (ablation) | PIPER · Maze navigation and robotic manipulation tasks | success rate | Mixed |
| Add hindsight relabeling (ablation) | PIPER · Sparse maze and kitchen | success rate | Helped |
| Add hindsight relabeling (ablation) | PIPER · Sparse pick and place and push | training speed | Helped |
Surprising findings
- PEBBLE with actual human preferences (PEBBLE-Dense) fails to show good performance in any task, even though PEBBLE works in simpler environments.
- Primitive-informed regularization gave only slight gains in maze, hollow and kitchen tasks, versus large gains in pick and place and push.
Limitations
- L2 distance on states as the informative metric may be hard to compute for high-dimensional goals such as images.
- Primitive-in-the-loop preferences may miss information human preferences carry, such as safety.
- Relies on a single human demonstration and an extra imitation objective at the lower level for the harder tasks.
Experimental setup
Datasets: simulated robotic environments (maze navigation, pick and place, push, hollow, franka kitchen)
Benchmarks: maze navigation pick and place push hollow franka kitchen
Metrics: success rate
Baselines: SAC-Dense PEBBLE-Dense RFLAT HIER HAC DAC FLAT SAC RAPS PIPER-No-V
Builds on: PEBBLE Hindsight Experience Replay Soft Actor-Critic HAC (Hierarchical Actor-Critic) RAPS
Closest papers by meaning
- PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning (ICLR 2025 · similarity 0.91)
- CRISP: Curriculum-Inducing Primitive Informed Subgoal Prediction for Boosting Hierarchical Reinforcement Learning (AAAI 2026 · similarity 0.90)
- Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments (NeurIPS 2022 · similarity 0.89)
- Deep Reinforcement Learning from Hierarchical Preference Design (ICML 2025 · similarity 0.89)
- Learning Process Rewards via Visitation Matching for Efficient RL (NeurIPS 2026 · similarity 0.88)
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.