Research map / Reinforcement learning / Visual and representation RL
Efficient Skill Discovery via Regret-Aware Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RSD, a min-max regret-aware unsupervised skill discovery method that builds on METRA with a bounded temporal representation and a population of learnable skill generators that target high-regret skills.
Problem
Unsupervised skill discovery in RL is inefficient, especially in high-dimensional environments, because skills are sampled uniformly and ignore how converged the agent's policy strength is for each skill.
Approach
Frame skill discovery as a min-max game: the agent minimizes TD-estimated regret over skills while a population of skill-generator policies (GMM components) is trained to maximize regret, with a bounded representation, a non-unit skill vector, and KL/seen-state regularizers.
Key results
- On zero-shot goal-reaching in Antmaze-large, RSD achieves AR 0.507 and FD 6.42 versus METRA-d's 0.435 and 6.76.
- On zero-shot goal-reaching in Maze2d-large, RSD achieves AR 0.811 and FD 0.535 versus METRA-d's 0.782 and 0.658.
- In Kitchen (skill dim 24) at 400k steps, RSD completes 5.08±0.45 tasks versus METRA's 3.94±0.36.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use regret-aware skill discovery instead of METRA-d | RSD · Maze2d-large | final distance (FD): 0.658 -> 0.535 | Helped |
| Use regret-aware skill discovery instead of METRA-d | RSD · Maze2d-large | success rate (AR): 0.782 -> 0.811 | Helped |
| Use regret-aware skill discovery instead of METRA-d | RSD · Antmaze-large | final distance (FD): 6.76 -> 6.42 | Helped |
| Use regret-aware skill discovery instead of METRA-f | RSD · Antmaze-large | success rate (AR): 0.450 -> 0.507 | Helped |
| Use RSD instead of METRA | Kitchen, skill dimension 24 · 400k steps | completed tasks: 3.94±0.36 -> 5.08±0.45 | Helped |
| Use RSD instead of METRA | Kitchen, skill dimension 32 · 400k steps | completed tasks: 3.99±0.94 -> 4.55±0.35 | Helped |
| Use RSD instead of METRA | Kitchen, skill dimension 24 · 100k steps | completed tasks: 2.72±0.19 -> 2.41±0.02 | Hurt |
Surprising findings
- The authors report RSD has lower state coverage than LSD and METRA in skill-asymmetric maze environments, attributing this to the broader skill space being learned within the same timesteps.
Limitations
- State coverage is lower than LSD and METRA in skill-asymmetric environments
- Zero-shot FD improvement is less pronounced in high-dimensional environments due to out-of-distribution goals
- Regret is estimated via TD error and can be biased, especially for unseen skills
Experimental setup
Datasets: D4RL maze2d-umaze-dense-v1 D4RL antmaze-medium-diverse-v0 D4RL antmaze-large-diverse-v0 dm_control Ant Kitchen (pixel-based)
Benchmarks: Maze2d-large Antmaze-medium Antmaze-large Ant (dm_control) Kitchen
Metrics: CoverCoords (unique coordinates visited) final distance (FD) success rate (AR) completed tasks
Baselines: METRA LSD DIAYN DADS
Builds on: METRA LSD SAC PLR PPO
Closest papers by meaning
- Constrained Ensemble Exploration for Unsupervised Skill Discovery (ICML 2024 · similarity 0.90)
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.90)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.88)
- Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions (ICML 2024 · similarity 0.88)
- No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.