Research map / Reinforcement learning / Visual and representation RL
Constrained Ensemble Exploration for Unsupervised Skill Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CeSD, an unsupervised skill discovery method that uses an ensemble of Q-networks, one per skill, each driven by particle-based entropy rewards within prototype-defined state clusters plus a distribution-constraint reward that discourages overlap between skills.
Problem
Unsupervised RL skill discovery: empowerment/MI-based methods yield static, low-coverage skills, while pure entropy exploration gives coverage without distinguishable useful behaviors.
Approach
Each skill gets its own Q-function trained only on transitions in one prototype-defined state cluster (Sinkhorn-Knopp clustering), with intrinsic particle-entropy reward inside that cluster, and an added reward penalizing visits to other skills' clusters. This bypasses MI estimation while keeping skills distinguishable.
Key results
- CeSD achieves 91.05% IQM on URLB, versus 75.18% for CIC.
- CeSD achieves the best optimality gap of 15.47%, compared with 25.44% for BeCL.
- In the maze, CeSD shows both higher skill discriminability and better state coverage than DIAYN, DADS, and CIC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CeSD partition exploration and state-distribution constraints instead of CIC global exploration | unsupervised skill pretraining · URLB 12 downstream tasks | IQM: 75.18% -> 91.05% | Helped |
| use CeSD instead of BeCL | unsupervised skill pretraining · URLB 12 downstream tasks | optimality gap: 25.44% -> 15.47% | Helped |
| reduce the ensemble value-function count while keeping the skill count unchanged (ablation) | CeSD · Quadruped downstream tasks | generalization performance | Hurt |
| set the distribution-constraint weight to α=0.1 (ablation) | CeSD · Maze | skill discriminability | Hurt |
| set the distribution-constraint weight to α∈[1.0,2.0] (ablation) | CeSD · Maze | skill discriminability | Helped |
| set the distribution-constraint weight to α≥10.0 (ablation) | CeSD · Maze | exploration | Hurt |
Surprising findings
- Reducing the ensemble size to 1 reduces CeSD to CIC-like behavior with indistinguishable skills.
- CIC obtains much lower mutual information than empowerment methods yet the largest state entropy.
Limitations
- Ensemble value functions do not generalize to continuous skill spaces.
- Overly large distribution-constraint weight alpha hinders exploration.
- The entropy-partition theorem relies on the assumption that clusters have equal sample counts.
Experimental setup
Datasets: URLB DeepMind Control Suite tasks (Walker, Quadruped, Jaco) 2D maze environment
Benchmarks: URLB (12 downstream tasks: Walker, Quadruped, Jaco) 2D maze
Metrics: IQM mean median optimality gap state coverage mutual information estimate
Baselines: DIAYN DADS SMM APS APT ProtoRL CIC ICM RND Disagreement BeCL
Builds on: DDPG BeCL CIC Sinkhorn-Knopp URLB
Closest papers by meaning
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.92)
- Behavior Contrastive Learning for Unsupervised Skill Discovery (ICML 2023 · similarity 0.91)
- Efficient Skill Discovery via Regret-Aware Optimization (ICML 2025 · similarity 0.90)
- Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Towards Principled Unsupervised Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.