Research map / Reinforcement learning / Visual and representation RL
Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes how MISL-learned skills initialize downstream task adaptation, proposes the LSEPIN disentanglement metric linked to worst-case adaptation cost, and a Wasserstein-based objective (WSEP) and algorithm (PWSEP) for discovering polytope vertices.
Problem
Mutual information skill learning (MISL) in unsupervised RL lacks analysis of how well learned skills initialize downstream task policies; it is unclear which skill properties (diversity, separability) govern adaptation cost.
Approach
Introduces LSEPIN (min over skills of I(S;1_z)) as a disentanglement metric tied to worst-case adaptation cost, and replaces KL with Wasserstein distance to define WSEP and PWSEP, which push skills to polytope vertices and can in theory discover all vertices.
Key results
- Theorem shows higher I(S;1_z) implies higher p(z) and lower adaptation cost IC_z for downstream tasks.
- Lemma shows that under stated assumptions, higher LSEPIN results in lower worst-case adaptation cost (WAC).
- PWSEP is theoretically shown to discover all polytope vertices with |V| skills; a small example shows WSEP finds all 5 vertices where MISL finds 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LSEPIN to MISL to favor skill sets with more separable and diverse skills | MISL · Downstream task adaptation | WAC | Helped |
| Replace KL divergence with Wasserstein distance and maximize pairwise skill distances (WSEP) | Unsupervised skill learning · Feasible state-distribution polytope | potentially optimal skills discovered | Helped |
| Use projected Wasserstein distance to the convex hull of previously discovered skills (PWSEP) | Unsupervised skill learning · Feasible state-distribution polytope | vertices discovered | Helped |
| Maximize WSEP | WSEP · Downstream task adaptation | MAC | Mixed |
| Replace the Wasserstein distances in WSEP with KL divergences | WSEP · Skill diversity and separability | diversity and separability | Hurt |
| Use uniform skill probabilities in MISL | MISL · Example with |S|=3 and 4 skills | skill diversity | Hurt |
Surprising findings
- MISL solutions with uniform skill prior need not yield separable or diverse skills.
- Maximizing WSEP can lower the adaptation cost bound not monotonically; more distant skills do not always yield lower adaptation cost.
Limitations
- Contribution is mainly theoretical; no empirical validation in this text (deferred to a companion paper).
- Wasserstein approach requires a transport cost that may need strong prior knowledge.
- Assumptions (far target distribution, large state space) needed to treat each skill's cost equally.
Experimental setup
Metrics: worst-case adaptation cost (WAC) mean adaptation cost (MAC)
Baselines: MISL SFOLS
Builds on: mutual information skill learning information geometry of MISL successor features (SFOLS)
Closest papers by meaning
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.90)
- Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.89)
- Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning (ICLR 2025 · similarity 0.88)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.87)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.