Research map / Reinforcement learning / Visual and representation RL
PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PEAR, which adaptively relabels a few expert demonstrations using the lower primitive's Q-function to build achievable subgoal data, then jointly optimizes off-policy SAC-based HRL with BC or GAIL-style IRL regularization.
Problem
Hierarchical RL with off-policy learning suffers from non-stationarity between levels, and fixed-window subgoal relabeling of expert demos yields subgoals that are too easy or too hard for the lower primitive, in sparse-reward long-horizon robotic tasks.
Approach
Use the current lower primitive's action-value function to adaptively select demonstration states as subgoals, periodically re-populating the subgoal dataset to form a curriculum of achievable subgoals, then regularize the joint RL objective with BC or IRL (GAIL with LSGAN) imitation terms, with a margin classification term to curb Q overestimation on out-of-distribution states.
Key results
- PEAR reaches up to 80% success on complex sparse robotic control tasks where baselines typically fail to show significant progress (5 seeds).
- On real-robot experiments, PEAR-IRL achieves success accuracy of 0.8 (pick and place), 0.6 (bin), and 0.3 (rope) over 5 sets of 10 trials.
- PEAR-IRL and PEAR-BC outperform PEAR-RPL (fixed-window relabeling) in almost all tasks, and the RPL baseline fails on all real-world tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed-window relabeling with adaptive relabeling (ablation) | PEAR-IRL · Maze navigation and manipulation tasks | success rate | Helped |
| add margin classification objective (ablation) | PEAR-IRL and PEAR-BC · Maze navigation and manipulation tasks | success rate | Helped |
| use IRL rather than BC regularization (ablation) | PEAR · Real-world pick and place, bin, and rope manipulation | accuracy: pick and place: 0.8 vs 0.8; bin: 0.6 vs 0; rope: 0.3 vs 0.3 | Mixed |
| use a larger number of sub-optimal demonstrations (ablation) | PEAR-IRL · Maze navigation and manipulation tasks | success rate | Hurt |
| vary the relabeling threshold Q_thresh (ablation) | PEAR · Six task environments | success rate | Mixed |
| vary the subgoal dataset re-population interval p (ablation) | PEAR · Six task environments | success rate | Mixed |
| vary fixed-window size k (ablation) | RPL · Six task environments | success rate | Mixed |
| vary IL regularization weight psi (ablation) | PEAR · Six task environments | success rate | Mixed |
| vary the number of expert demonstrations (ablation) | PEAR · Six task environments | success rate | Mixed |
| replace hierarchical PEAR-IRL with single-level HER-BC (ablation) | PEAR-IRL · Complex task environments | success rate | Helped |
| add expert demonstrations and hierarchical behavior cloning (ablation) | HAC and hierarchical baselines · Six task environments | success rate | Mixed |
| remove margin classification objective (ablation) | PEAR-IRL and PEAR-BC · Six task environments | training stability | Hurt |
Surprising findings
- PEAR-BC achieves 0 success on the real-world bin task while PEAR-IRL achieves 0.6.
Limitations
- Requires availability of directed expert demonstrations.
- Periodic re-population of the subgoal dataset adds overhead that may be a bottleneck when relabeling is costly.
- Real-world experiments use hard-coded velocity values and state estimates from a depth camera rather than accurate velocity estimation.
Experimental setup
Datasets: expert demonstrations generated by RRT (maze), hard-coded policies (pick and place, bin, hollow), VR teleoperation (kitc
Benchmarks: maze navigation pick and place bin hollow rope manipulation franka kitchen
Metrics: success rate average distance between predicted and achieved subgoals
Baselines: RPL RAPS HAC HIER HIER-NEG DAC SAC (FLAT) BC HER-BC
Builds on: Soft Actor-Critic Relay Policy Learning Hierarchical Actor-Critic GAIL Hindsight Experience Replay
Closest papers by meaning
- PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling (ICML 2024 · similarity 0.91)
- CRISP: Curriculum-Inducing Primitive Informed Subgoal Prediction for Boosting Hierarchical Reinforcement Learning (AAAI 2026 · similarity 0.90)
- Data-Efficient Hierarchical Reinforcement Learning (NeurIPS 2018 · similarity 0.90)
- Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives (NeurIPS 2021 · similarity 0.88)
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach (ICLR 2026 · similarity 0.88)
- An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.