Research map / Reinforcement learning / Policy gradient and actor-critic

Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Constrained policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GowU, a Go-With-The-Winner particle tree search guided by RND epistemic uncertainty that explores hard Atari and MuJoCo tasks without RL, then distills the trajectories into policies via a backward PPO curriculum.

Problem

Autonomous exploration in sparse-reward settings is inefficient when intrinsic motivation is maximized through RL policy optimization; hard-exploration Atari games and high-dimensional continuous control from pixels remain unsolved.

Approach

Decouple exploration from policy optimization: a population of particles performs uncertainty-guided GWTW-style tree search with resets to winners (max reward, tie-broken by RND uncertainty) and rollbacks, using random policies; trajectories are then distilled into deployable policies by backward learning.

Key results

• GowU Phase II policy scores: Montezuma 196,312 ± 49,670, Pitfall 101,972 ± 2,226, Venture 6,436 ± 1,504, versus Go-Explore 43,791 / 6,945 / 2,281 on the same games (Go-Explore exploration baseline numbers).
• Phase II success rates from pixels with sparse rewards: hammer 100.0%, door 98.1%, relocate 95.2%, AntMaze 96.1%.
• Phase I exploration on Atari reaches 98,249 mean cumulative reward on Montezuma's Revenge within 400M frames, and the paper claims an order-of-magnitude efficiency gain over intrinsic-motivation baselines.

Surprising findings

• Random action policies within a tree search suffice to discover high-reward trajectories, with no RL during exploration.
• Distilled policies from backward learning from checkpoints spaced several inner steps apart still perform well.

Limitations

• Relies on the ability to reset the environment to previously visited states (simulator access during training).
• Requires hand-designed dead-state criteria and intermediate rewards for Phase I on several environments.
• RND may assign persistently high novelty to stochastic observations.

Experimental setup

Datasets: Montezuma's Revenge (ALE) Pitfall! (ALE) Venture (ALE) MuJoCo Adroit door/hammer/relocate MuJoCo AntMaze

Benchmarks: Montezuma's Revenge Pitfall! Venture Adroit door Adroit hammer Adroit relocate AntMaze

Metrics: cumulative reward policy score success rate frames to first task completion

Baselines: Go-Explore RND MEME BYOL-Hindsight

Builds on: Go-With-The-Winner Go-Explore Random Network Distillation backward learning curriculum PPO

Closest papers by meaning

• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Exploration via Planning for Information about the Optimal Trajectory (NeurIPS 2022 · similarity 0.90)
• Generative Exploration and Exploitation (AAAI 2020 · similarity 0.90)
• Fast active learning for pure exploration in reinforcement learning (ICML 2021 · similarity 0.90)
• Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.89)
• MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.89)