Research map / Reinforcement learning / Visual and representation RL

Exploration by Learning Diverse Skills through Successor State Representations

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Unsupervised skill discovery · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LEADS, a reward-free skill discovery method that maximizes a variant of mutual information between states and skills, using C-Learning successor state representations and a novelty-based uncertainty measure to pick exploration target states, and reports higher state coverage than DIAYN, SMM, LSD, CSD, METRA, RND and NGU on maze and MuJoCo tasks.

Problem

Unsupervised skill discovery in reward-free RL: standard mutual information objectives yield diverse skills that do not necessarily cover the state space, so exploration is poor.

Approach

Replaces the plain MI objective with a bound on the ratio of each skill's SSR to the summed SSRs of all skills, and biases each skill's sampling toward states chosen by a successor-state-based uncertainty measure that combines an under-visited term with inter-skill repulsion terms.

Key results

• LEADS reaches 90.8% coverage on the Hard maze versus 70.0% for LSD and 64.0% for CSD (final coverage, five seeds).
• On Fetch-Slide LEADS reaches 85.4% versus 63.0% for CSD, exceeding all other methods by over 20%.
• CSD achieves 96.2% on Finger, above LEADS at 87.4%, so LEADS is not best on every task.

Measured effects reported in the paper

Surprising findings

• C-Learning-based SSR estimates fail on HalfCheetah MuJoCo, so LEADS performs poorly there despite extensive hyperparameter search.

Limitations

• Performance depends on the quality of the SSR estimator; C-Learning fails on some MuJoCo tasks such as HalfCheetah.
• The number of skills is hand-chosen and fixed (6 or 12) rather than adapted.
• The deterministic Dirac choice of exploration target was found empirically effective and is not theoretically justified; the lower bound guarantee is lost when sampling from delta.

Experimental setup

Datasets: Easy maze U maze Hard maze Fetch-Reach Fetch-Slide Hand Finger

Benchmarks: Easy maze U maze Hard maze Fetch-Reach Fetch-Slide Finger

Metrics: relative state coverage (%) discretized visited-cell count paired t-test p-values

Baselines: DIAYN SMM LSD CSD METRA RND NGU

Builds on: DIAYN C-Learning State Marginal Matching LSD successor features

Closest papers by meaning

• A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.89)
• Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.89)
• Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Periodic Skill Discovery (NeurIPS 2025 · similarity 0.87)
• Language Guided Skill Discovery (ICLR 2025 · similarity 0.87)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts (ICML 2024 · similarity 0.87)