Research map / Reinforcement learning / Visual and representation RL
Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUSDi, an MI-based unsupervised skill discovery method that learns disentangled skill components each affecting one state factor, using Q-decomposition, and uses them as low-level policies for hierarchical RL downstream.
Problem
Unsupervised skill discovery methods learn entangled skills where one skill variable affects many state factors, making downstream skill chaining hard for hierarchical RL.
Approach
Factorize the skill latent into per-state-factor components and reward each component for high mutual information with its assigned state factor and low mutual information with all other factors; decompose the Q-function per factor for faster, more stable learning.
Key results
- DUSDi achieves DCI disentanglement 0.864 on 2D Gunner vs 0.016 for DIAYN-MC, and 0.705 vs 0.002 on Multi-Particle.
- DUSDi significantly outperforms DIAYN and other unsupervised RL baselines on downstream tasks requiring coordinated control of multiple state factors, while performing similarly on DMC Walker.
- With image observations (object-centric encoder) on Multi-Particle, DUSDi achieves similar performance to state-based learning while baselines fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · 2D Gunner | Disentanglement: 0.016 ± 0.002 -> 0.864 ± 0.018 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · Multi-Particle | Disentanglement: 0.002 ± 0.000 -> 0.705 ± 0.037 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · iGibson | Disentanglement: 0.017 ± 0.006 -> 0.833 ± 0.022 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · 2D Gunner | Completeness: 0.024 ± 0.004 -> 0.864 ± 0.017 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · Multi-Particle | Completeness: 0.003 ± 0.000 -> 0.750 ± 0.041 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · iGibson | Completeness: 0.019 ± 0.005 -> 0.834 ± 0.021 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · 2D Gunner | Informativeness: 0.821 ± 0.010 -> 0.897 ± 0.012 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · Multi-Particle | Informativeness: 0.791 ± 0.032 -> 0.849 ± 0.052 | Helped |
| Use DUSDi's disentangled skill objective instead of DIAYN-MC's DIAYN objective | DIAYN-style skill discovery · iGibson | Informativeness: 0.752 ± 0.015 -> 0.854 ± 0.006 | Helped |
| Add Q-decomposition to the DUSDi skill learner (ablation) | DUSDi · 2D Gunner | Skill-discriminator accuracy | No clear effect |
| Add Q-decomposition to the DUSDi skill learner (ablation) | DUSDi · Multi-Particle | Skill-discriminator accuracy | Helped |
| Add Q-decomposition to the DUSDi skill learner (ablation) | DUSDi · iGibson | Skill-discriminator accuracy | Helped |
Surprising findings
- Increasing value-network capacity for baselines did not significantly improve skill or task learning, suggesting disentangled skills rather than capacity drive the gains.
- CIC performs very poorly on downstream tasks because its objective does not encourage distinguishable skills.
Limitations
- Assumes access to a factored state space; image-based settings rely on a separately pretrained object-centric encoder.
- Exploration during skill learning depends on the underlying MI-based algorithm (DIAYN); only DIAYN-style objective is used.
- Lambda < 1 trade-off needed for environments with intrinsic dependencies between state factors; gains are smaller in DMC Walker where downstream tasks need only center-of-mass control.
Experimental setup
Benchmarks: DMC Walker 2D Gunner Multi-Particle iGibson
Metrics: DCI disentanglement DCI completeness DCI informativeness classification accuracy of skill discriminators downstream task return
Baselines: DIAYN CIC CSD METRA ICM RND ELDEN SAC
Builds on: DIAYN SAC PPO Causal Policy Gradient
Closest papers by meaning
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.92)
- SUSD: Structured Unsupervised Skill Discovery through State Factorization (ICLR 2026 · similarity 0.92)
- Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Constrained Ensemble Exploration for Unsupervised Skill Discovery (ICML 2024 · similarity 0.89)
- SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions (NeurIPS 2024 · similarity 0.88)
- Efficient Skill Discovery via Regret-Aware Optimization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.