Research map / Reinforcement learning / Visual and representation RL
Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hyper, a curiosity-driven RL method that decouples exploitation from exploration via a repositioning phase (bounded geometric length) before exploration, yielding robustness to the curiosity coefficient β and a provable efficiency bound for linear MDPs.
Problem
Curiosity-driven exploration methods are highly sensitive to the intrinsic reward coefficient β, requiring extensive per-environment tuning, and large β causes over-exploration and optimization instability.
Approach
Hyper runs a repositioning phase (following the exploitation policy for a bounded-geometric number of steps) before a curiosity-driven exploration phase, with a separate exploitation policy learned off-policy from all data, so exploration visitation is regularized and persistent and large β becomes tolerable.
Key results
- Hyper is the only agent that consistently solves the hardest navigation task LargeMaze-Hard, while Decouple fails there.
- Curiosity's performance peaks at small beta and drops sharply for beta > 10, whereas Hyper remains consistent across beta values (5 runs).
- Theory: Linear-UCB-Hyper obtains an epsilon-optimal exploitation policy within O~(d^3 H^4 / epsilon^2) steps with high probability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hyper instead of Decouple | LargeMaze-Hard · LargeMaze-Hard | success rate | Helped |
| Use Hyper instead of Curiosity | curiosity-driven exploration · Five environments in β-sensitivity analysis | final performance across β values | Helped |
| Use Curiosity instead of TD3 | TD3 · Hopper-Sparse and Walker2d-Sparse | episodic cumulative reward | Helped |
| Use Hyper instead of Curiosity | curiosity-driven exploration · Dense-reward locomotion tasks | episodic cumulative reward | No clear effect |
| Use Hyper instead of Decouple | decoupled reinforcement learning · High-dimensional locomotion tasks | episodic cumulative reward | Helped |
Surprising findings
- Decoupled exploitation can suffer distribution shift and fail on high-dimensional locomotion tasks even when exploration data is collected by a curiosity agent.
Limitations
- Theoretical guarantee is for linear MDPs with UCB-style value iteration, not the neural empirical version.
- Decouple baseline struggles in high-dimensional locomotion, and Hyper's gains are demonstrated on a limited set of environments.
Experimental setup
Datasets: PointMaze (MediumMaze, LargeMaze) MuJoCo locomotion (Hopper, Walker2d, HalfCheetah, Ant, Humanoid) with sparse variants MiniGrid
Benchmarks: PointMaze MediumMaze and LargeMaze navigation tasks MuJoCo sparse and dense locomotion MiniGrid (Empty16x16, Fetch8x8, UnlockPickup, LavaCrossingS9N1)
Metrics: final success rate episodic cumulative reward sensitivity across beta values
Baselines: TD3 Curiosity (Disagreement-driven TD3) Decouple LESSON UCB-Q
Builds on: TD3 Disagreement intrinsic reward Decoupled Reinforcement Learning UCB-Q LSVI-UCB
Closest papers by meaning
- Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent (ICML 2024 · similarity 0.88)
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.87)
- Parameter Exploration for RLVR via Variational Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.