Research map / Reinforcement learning / Visual and representation RL

DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: Reward shaping for sparse rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DISCOVER, a goal-selection method for sparse-reward goal-conditioned RL that picks achievable, novel, and target-relevant exploratory goals using an ensemble of TD3 critics, with a regret-style bound independent of goal-space volume.

Problem

Exploration in sparse-reward RL for single very hard, high-dimensional, long-horizon target tasks, where undirected goal-conditioned exploration becomes intractable.

Approach

Select exploratory goals from achieved goals by maximizing a UCB-style objective combining achievability V(s0,g), novelty sigma(s0,g), and relevance V(g,g*) estimated from an ensemble of critics, so exploration is bootstrapped toward the target without prior information; an online rule adapts the trade-off to a 50% goal achievement rate.

Key results

• DISCOVER reaches 10% target achievement in pointmazes up to 6 dimensions (e.g., 18.7M steps at dimension 6), whereas HER, MEGA and Achievability+Novelty fail (infinite) beyond dimension 2-3.
• DISCOVER consistently outperforms goal-selection baselines across pointmaze, antmaze and arm environments, learning difficult target tasks significantly faster.
• State-of-the-art non-goal-conditioned exploration methods (curiosity-based, e.g. MaxInfoRL with SAC) fail to reach the target task within the maximum episodes even in simple configurations.

Measured effects reported in the paper

Surprising findings

• In the arm environment, HER outperforms some undirected goal-selection strategies, which the authors attribute to irrelevant exploration in an unconstrained goal space.
• Adding a sigma(g, g*) uncertainty term for relevance showed no significant positive effect on performance, so it was dropped in other experiments.

Limitations

• Relies on bootstrapped value estimates whose quality depends on value-network generalization.
• Training an ensemble of critics adds overhead, limiting use with large critic networks such as language models.
• Theory holds only in a simplified linear bandit setting with strong assumptions (linear value features, achievability, geodesic convexity).

Experimental setup

Datasets: JaxGCRL pointmaze JaxGCRL antmaze JaxGCRL arm randomly generated n-dimensional pointmazes

Benchmarks: pointmaze (2-6 dimensions) antmaze (simple and hard) arm (simple and hard)

Metrics: success rate on target goal environment steps to 10% target goal achievement goal achievement rate

Baselines: HER DISCERN (uniform) MEGA Achievability + Novelty MaxInfoRL (SAC)

Builds on: TD3 Hindsight Experience Replay MEGA DISCERN UCB

Closest papers by meaning

• First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs (NeurIPS 2024 · similarity 0.90)
• Learning Process Rewards via Visitation Matching for Efficient RL (NeurIPS 2026 · similarity 0.90)
• Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.90)
• A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.90)
• MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.89)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)