Research map / Reinforcement learning / Visual and representation RL
Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Subwords as Skills (SaS), which discretizes demonstration actions with k-means and applies BPE to extract an unconditioned skill vocabulary for sparse-reward online RL with SAC-discrete.
Problem
Exploration in sparse-reward RL requires long coordinated action sequences, but existing skill-extraction methods from demonstrations are expensive to train and slow to run at inference.
Approach
Treat demonstration action sequences as strings over k-means action tokens and apply byte-pair encoding to extract a finite vocabulary of observation-free skills, pruned to a fixed skill length and vocabulary size, which then serves as the action space for SAC-discrete.
Key results
- On AntMaze Medium, SaS skill generation takes about 3 seconds versus ~130000 s for SSP and ~8000 s for SFP, with a 0.007 s online rollout versus 0.9 s for SSP and 4.1 s for SFP.
- SaS strongly outperforms unconditional baselines (SAC, SAC-discrete, SSP, SFP) on AntMaze and Kitchen, where baselines mostly achieve 0 return, and is competitive on CoinRun.
- Skills extracted from 1% of AntMaze-M demonstrations transfer to AntMaze-U with stable performance down to 10 trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SaS instead of SSP for skill extraction | AntMaze Medium skill extraction · AntMaze Medium | skill generation time: 130000 ± 1800 -> 3 ± 1 | Helped |
| Use SaS instead of SFP for skill extraction | AntMaze Medium skill extraction · AntMaze Medium | skill generation time: 8000 ± 500 -> 3 ± 1 | Helped |
| Use SaS instead of SSP for online rollouts | AntMaze Medium online RL · AntMaze Medium | online rollout time: 0.9 ± 0.05 -> 0.007 ± 0.0006 | Helped |
| Use SaS instead of SFP for online rollouts | AntMaze Medium online RL · AntMaze Medium | online rollout time: 4.1 ± 0.1 -> 0.007 ± 0.0006 | Helped |
| Use more clusters than the default (ablation) | SaS · AntMaze | RL performance | No clear effect |
| Use fewer clusters than the default (ablation) | SaS · AntMaze and Kitchen | RL performance | No clear effect |
| Set subword length to 5 instead of the default (ablation) | SaS · AntMaze | RL performance | Hurt |
| Vary subword length from the default (ablation) | SaS · Kitchen | RL performance | No clear effect |
| Increase vocabulary size above the default (ablation) | SaS · AntMaze | RL performance | Hurt |
| Replace BPE with WordPiece (ablation) | SaS · AntMaze and Kitchen | RL performance | No clear effect |
| Replace BPE with Unigram (ablation) | SaS · AntMaze and Kitchen | RL performance | Hurt |
| Increase action discretization resolution (ablation) | SaS · Hopper | return | Mixed |
Surprising findings
- Random Hopper demonstrations yield quite competitive skills, which the authors attribute to short subwords still enabling dense-reward recovery.
- Observation-conditioned SPiRL performs very strongly on Kitchen but struggles on AntMaze, where the demonstrations are disjoint from the task.
Limitations
- Discretization removes action resolution, which hurts fast locomotion (Hopper experiments).
- Subword execution is open-loop, which may limit exploration efficiency and safety.
- Large standard deviations because only five seeds were run and some seeds fail to obtain any reward.
Experimental setup
Datasets: D4RL AntMaze demonstrations D4RL Kitchen demonstrations CoinRun demonstrations D4RL Hopper
Benchmarks: AntMaze Kitchen CoinRun Hopper
Metrics: return skill generation time (s) online rollout time (s) state visitation
Baselines: SAC SAC-discrete Skill-Space Policy State-Free Priors SPiRL OPAL
Builds on: byte-pair encoding SAC-discrete Skill-Space Policy State-Free Priors
Closest papers by meaning
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.87)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.87)
- When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.