Research map / Reinforcement learning / Visual and representation RL
Cluster-Based Sampling in Hindsight Experience Replay for Robotic Tasks (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HER-CS, which clusters episodes by their last achieved goals (cluster model fit on failed goals) and samples episodes uniformly per cluster before applying HER, improving sample efficiency on three OpenAI Gym robotic tasks.
Problem
Sparse binary reward multi-goal RL suffers from few successful experiences; HER's uniform episode sampling is inefficient at producing useful hindsight experiences.
Approach
Replace HER's first uniform episode-sampling step with cluster-based sampling: group episodes by last achieved goal using a cluster model trained on failed goals, then sample uniformly across clusters so that hard episodes are more likely selected; the cluster model is periodically updated.
Key results
- HER-CS improves convergence speed on the Push task and maximum success rate on PickAndPlace over HER.
- Applying cluster-based sampling to HER-EBP (HER-EBP-CS) also yields performance gains.
- All four algorithms reach comparable success rates on the Slide task, with only marginal improvements.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cluster-based episode sampling (HER-CS) instead of uniform episode sampling | HER · Push | convergence speed | Helped |
| Use cluster-based episode sampling (HER-CS) instead of uniform episode sampling | HER · PickAndPlace | maximum success rate | Helped |
| Use cluster-based episode sampling (HER-CS) instead of uniform episode sampling | HER · Slide | success rate | No clear effect |
| Use cluster-based episode sampling (HER-EBP-CS) instead of the baseline sampling strategy | HER-EBP · Push | convergence speed | Helped |
| Use cluster-based episode sampling (HER-EBP-CS) instead of the baseline sampling strategy | HER-EBP · PickAndPlace | maximum success rate | Helped |
| Use cluster-based episode sampling (HER-EBP-CS) instead of the baseline sampling strategy | HER-EBP · Slide | success rate | No clear effect |
Surprising findings
- On the Slide task all methods perform comparably, indicating inherent difficulty limiting gains.
Limitations
- Gains are marginal on the Slide task.
- Results reported over only five random seeds and 20 test episodes per evaluation.
- Cluster update frequency and number of failed goals are hyperparameters studied only in the supplement.
Experimental setup
Datasets: OpenAI Gym robotic environments (Push, PickAndPlace, Slide)
Benchmarks: Push PickAndPlace Slide
Metrics: success rate
Baselines: HER HER-EBP
Builds on: hindsight experience replay energy-based hindsight experience prioritization
Closest papers by meaning
- Hindsight Experience Replay (NeurIPS 2017 · similarity 0.89)
- Exploration via Hindsight Goal Generation (NeurIPS 2019 · similarity 0.88)
- Goal-conditioned Imitation Learning (NeurIPS 2019 · similarity 0.88)
- Goal-Conditioned On-Policy Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- How Does Goal Relabeling Improve Sample Efficiency? (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.