Research map / Reinforcement learning / Visual and representation RL

A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: Novelty and impact-driven exploration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that contrastive RL trained with data collected only toward a single hard goal (no rewards, demonstrations, or subgoals) learns skills and directed exploration and solves manipulation and maze goal-reaching tasks.

Problem

Goal-conditioned RL with only a single difficult goal state provided, where random exploration never reaches the goal and no reward, demonstrations, or subgoal curriculum are available.

Approach

Use contrastive RL (infoNCE critic with inner-product representations) and always condition data collection on the single target goal, without any changes to the critic or actor losses; skills and directed exploration emerge without rewards or subgoals.

Key results

• Single-goal contrastive RL achieves first success after about 3,197 trials (sawyer box), 9,329 (sawyer bin), 15,895 (sawyer peg), and 11,724 (point spiral), while a uniform random policy never succeeded in 75M steps.
• Single-goal CRL significantly outperforms the range-of-difficulties multi-goal oracle and reward-based baselines (SAC variants, RIS), which rarely reach the goal except RIS on the point spiral task.
• Using a monolithic critic instead of the inner-product contrastive critic makes single-goal exploration ineffective, and training the actor only on the single goal degrades performance.

Measured effects reported in the paper

Surprising findings

• Single-goal exploration works even though the value function predicts uninformative values for the unreached goal early in training, contrary to the authors' initial intuition.
• Commanding an impossible goal in a maze makes the agent navigate toward it and get stuck, yet still provides some exploration.

Limitations

• No theoretical analysis explaining why skills and directed exploration emerge.
• Experiments focus primarily on manipulation tasks and one maze environment.
• Exploration plateaus after success, and the impossible-goal approach fails to explore many states.

Experimental setup

Datasets: sawyer box sawyer bin sawyer peg point spiral

Benchmarks: sawyer box sawyer bin sawyer peg point spiral

Metrics: success rate trials to first success unique grid cells visited robustness under static and dynamic perturbations

Baselines: RIS SAC with sparse rewards SAC+HER with sparse rewards SAC with dense rewards multi-goal contrastive RL (range of difficulties)

Builds on: contrastive RL SAC RIS HER

Closest papers by meaning

• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions (ICML 2024 · similarity 0.89)
• Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.89)
• Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.89)
• First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs (NeurIPS 2024 · similarity 0.89)
• Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.89)