Research map / Reinforcement learning / Visual and representation RL
Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUPE, which pretrains trajectory-segment VAE skills offline and reuses the unlabeled prior data online by pseudo-labeling it with skill latents and UCB (RND-based) rewards for a high-level SAC policy.
Problem
Online exploration in long-horizon, sparse-reward RL is slow; prior methods use unlabeled offline trajectories either only for offline skill pretraining or only as online off-policy data, but not both.
Approach
Use unlabeled prior trajectories twice: pretrain a trajectory-segment VAE skill space offline, then pseudo-label the offline segments with inferred skill latents and optimistic UCB rewards (reward model plus RND bonus) so they serve as high-level off-policy data for online RL over the frozen skills.
Key results
- SUPE outperforms prior baselines on most of 8 domains (42 tasks), except antsoccer and scene where SUPE (HILP) performs best and cube-single where ExPLORe is competitive.
- On antmaze, SUPE reaches goals faster than every baseline on all three maze layouts in terms of environment steps to first goal.
- On humanoidmaze, SUPE is the only method to achieve nonzero final return on the large and giant mazes; RND coefficient alpha=0 degrades performance significantly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unlabeled offline data during online learning in addition to skill pretraining | SUPE · eight evaluated domains | normalized return | Helped |
| pretrain skills from unlabeled offline data | SUPE · eight evaluated domains | normalized return | Helped |
| use SUPE with HILP skills instead of trajectory-VAE skills | SUPE · AntSoccer and Scene | normalized return | Helped |
| use trajectory-VAE skills instead of HILP skills | SUPE · domains other than AntSoccer and Scene | normalized return | Helped |
| add an RND bonus when pseudo-labeling offline data (ablation) | SUPE · representative state-based tasks | normalized return | Helped |
| vary the RND coefficient among 2, 8, and 16 (ablation) | SUPE · seven representative tasks | normalized return | No clear effect |
| use skill horizon 4 instead of horizon 2 (ablation) | SUPE · AntMaze large top-right goal | normalized return | Mixed |
| use skill horizon 4 instead of horizon 2 (ablation) | SUPE · tasks other than AntMaze large top-right goal | normalized return | Helped |
| use skill horizon 8 instead of horizon 4 (ablation) | SUPE · AntSoccer and Kitchen | normalized return | Helped |
| use skill horizon 8 instead of horizon 4 (ablation) | SUPE · other evaluated tasks | normalized return | Hurt |
| replace entropy regularization with a KL penalty to the state-dependent prior (ablation) | SUPE · AntMaze | normalized return | Hurt |
| initialize the RND image encoder with ICVF weights (ablation) | SUPE · Visual AntMaze | normalized return | Helped |
Surprising findings
- Removing the RND optimism bonus (alpha=0) makes SUPE perform significantly worse, highlighting the importance of optimistic labeling.
- Without ICVF initialization, SUPE on visual-antmaze still outperforms baselines that use ICVF.
Limitations
- Pretrained skills remain frozen during online learning, which can hinder learning when skills are poorly learned.
- Reliance on RND for the upper-confidence bound may need more careful design in other high-dimensional environments.
- Sensitive to skill horizon length; horizon 4 works best but shorter or longer horizons perform worse.
Experimental setup
Datasets: D4RL antmaze D4RL kitchen OGBench humanoidmaze OGBench antsoccer OGBench cube OGBench scene visual-antmaze
Benchmarks: D4RL antmaze D4RL kitchen OGBench humanoidmaze OGBench antsoccer OGBench cube-single OGBench cube-double OGBench scene visual-antmaze
Metrics: normalized return interquartile mean IQM of environment steps to first goal exploration coverage
Baselines: ExPLORe DBC+JSRL Trajectory Skills HILP Skills SUPE (HILP) Online Only Ours (KL)
Builds on: OPAL ExPLORe RLPD SPiRL RND
Closest papers by meaning
- Accelerating Exploration with Unlabeled Prior Data (NeurIPS 2023 · similarity 0.92)
- Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data (ICLR 2026 · similarity 0.91)
- How to Leverage Unlabeled Data in Offline Reinforcement Learning (ICML 2022 · similarity 0.91)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.88)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.