Research map / Reinforcement learning / Visual and representation RL
Periodic Skill Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Periodic Skill Discovery (PSD), which learns a circular latent space encoding temporal distance so that a single-step reward yields periodic skills with diverse periods, and combines it with METRA for fully unsupervised skill discovery.
Problem
Existing unsupervised skill discovery methods (MI-based and distance-maximizing) ignore the periodic structure of behaviors, so they fail to discover diverse periodic skills across multiple timescales for locomotion.
Approach
An encoder maps states conditioned on a period variable L to a circular latent of diameter L, with constraints that L-step states are antipodal and adjacent states are spaced at angle pi/L; a single-step reward penalizes deviation from the optimal adjacent-step latent length, and an adaptive sampling scheme expands the period range.
Key results
- On downstream hurdle and friction tasks over 10 seeds, PSD reaches e.g. 3.8 on HalfCheetah-hurdle vs 1.9 for METRA and 43.4 on HalfCheetah-friction vs 30.1 for METRA.
- PSD learns periodic skills in pixel-based Ant and HalfCheetah environments without modifying its objective.
- Empirically, 1-step and L-step latent distances converge close to theoretical optima L sin(pi/2L) and L with small relative error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSD skills instead of METRA skills | HalfCheetah-hurdle high-level policy · HalfCheetah-hurdle | average return: 1.9 ±0.8 -> 3.8 ±2.0 | Helped |
| Use PSD skills instead of CSD skills | Walker2D-hurdle high-level policy · Walker2D-hurdle | average return: 4.1 ±1.3 -> 5.4 ±1.4 | Helped |
| Use PSD skills instead of METRA skills | HalfCheetah-friction high-level policy · HalfCheetah-friction | average return: 30.1 ±13.1 -> 43.4 ±19.1 | Helped |
| Use PSD skills instead of CSD skills | Walker2D-friction high-level policy · Walker2D-friction | average return: 5.3 ±0.3 -> 8.7 ±1.7 | Helped |
| Combine PSD and METRA with independently trained encoders and summed intrinsic rewards, without mutual conditioning (ablation) | PSD combined with METRA · MuJoCo locomotion | skill diversity | Hurt |
| Mutually condition the PSD and METRA encoders on the other method’s skill variable (ablation) | PSD combined with METRA · MuJoCo locomotion | skill diversity | Helped |
| Use a 2-dimensional latent space instead of the reported 3- or 6-dimensional spaces (ablation) | PSD on complex agents · Ant and Humanoid | training stability | Hurt |
Surprising findings
- A naive independent combination of PSD and METRA collapses all skills to a single short-period behavior, requiring mutual conditioning of encoders to avoid collapse.
Limitations
- May underperform under large persistent external disturbances where periodic behavior is infeasible.
- Focused on locomotion tasks; extension to non-periodic tasks such as manipulation needs generalized latent geometry.
- A 2-dimensional latent space led to unstable performance for Ant and Humanoid.
Experimental setup
Datasets: MuJoCo locomotion environments (Ant, HalfCheetah, Humanoid, Hopper, Walker2D) downstream hurdle and friction tasks
Benchmarks: MuJoCo Ant MuJoCo HalfCheetah MuJoCo Humanoid MuJoCo Hopper MuJoCo Walker2D HalfCheetah-hurdle Walker2D-hurdle HalfCheetah-friction Walker2D-friction
Metrics: average return frequency spectrum dominance of skill trajectories 1-step and L-step latent distance error
Baselines: DIAYN DADS CSD METRA
Builds on: DIAYN METRA SAC PPO CURL
Closest papers by meaning
- SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions (NeurIPS 2024 · similarity 0.89)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.88)
- Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.87)
- METRA: Scalable Unsupervised RL with Metric-Aware Abstraction (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.