Research map / Reinforcement learning / Visual and representation RL
SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkillTree, a hierarchical RL framework where a differentiable soft decision tree acts as a high-level policy over a discrete VQ-VAE skill codebook, then distills it into a small hard decision tree for explainable long-horizon robotic control.
Problem
Decision-tree-based RL policies are hard to optimize and lack expressiveness for long-horizon, high-dimensional, continuous-control tasks, so explainable RL has been limited to simple environments.
Approach
Learn a discrete skill codebook from task-agnostic offline data with a modified VQ-VAE and skill prior, then train the high-level policy as a soft decision tree that selects discrete skills, followed by CART-based distillation into a hard decision tree.
Key results
- SkillTree (DC + Distilling) reaches 3.25 ACS on Kitchen MKBL and 3.00 ACS on CALVIN, versus 2.97-3.08 for SPiRL/VQ-SPiRL on Kitchen MKBL and 2.77 for SPiRL on CALVIN.
- CART baseline achieves only 2.11 ACS on Kitchen MKBL and 0.00 on Office, with about 950-1000 leaf nodes versus 64 for SkillTree.
- BC+SAC completes no subtasks in any domain because of sparse rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the soft decision tree (ablation) | SkillTree · Kitchen MKBL | ACS: 2.90 $\pm$ 0.42 -> 3.06 $\pm$ 0.65 | Helped |
| distill the soft decision tree (ablation) | SkillTree · Kitchen MLSH | ACS: 2.58 $\pm$ 0.80 -> 2.52 $\pm$ 0.82 | Hurt |
| distill the soft decision tree (ablation) | SkillTree · CALVIN | ACS: 1.90 $\pm$ 0.90 -> 2.42 $\pm$ 0.49 | Helped |
| distill the soft decision tree (ablation) | SkillTree · Office | ACS: 2.00 $\pm$ 0.00 -> 2.00 $\pm$ 0.00 | No clear effect |
| clean sampled trajectories, then distill the soft decision tree (ablation) | SkillTree · Kitchen MKBL | ACS: 2.90 $\pm$ 0.42 -> \textbf{3.25} $\pm$ \textbf{0.70} | Helped |
| clean sampled trajectories, then distill the soft decision tree (ablation) | SkillTree · Kitchen MLSH | ACS: 2.58 $\pm$ 0.80 -> 2.62 $\pm$ 0.76 | Helped |
| clean sampled trajectories, then distill the soft decision tree (ablation) | SkillTree · CALVIN | ACS: 1.90 $\pm$ 0.90 -> \textbf{3.00} $\pm$ \textbf{0.00} | Helped |
| clean sampled trajectories, then distill the soft decision tree (ablation) | SkillTree · Office | ACS: 2.00 $\pm$ 0.00 -> \textbf{2.00} $\pm$ \textbf{0.00} | No clear effect |
| use a discrete skill representation and decision-tree high-level policy instead of SPiRL | SkillTree · Office | ACS: 1.48 $\pm$ 0.96 -> 2.00 $\pm$ 0.00 | Helped |
| use a discrete skill representation and decision-tree high-level policy instead of VQ-SPiRL | SkillTree · Office | ACS: 1.69 $\pm$ 0.64 -> \textbf{2.00} $\pm$ \textbf{0.00} | Helped |
Surprising findings
- SPiRL achieves the best ACS on Kitchen MLSH (2.82) over SkillTree variants, and SkillTree is roughly comparable or lower on several domains, so it does not uniformly outperform neural baselines.
- Office results for all SkillTree variants are a flat 2.00 with zero variance, suggesting a ceiling effect in that domain.
Limitations
- Low-level policy remains a neural network and is not explainable.
- Requires an offline task-agnostic dataset for skill learning, which may not be available in practice.
- Does not uniformly beat neural skill-based baselines (e.g., SPiRL on Kitchen MLSH).
Experimental setup
Datasets: D4RL Franka Kitchen offline dataset CALVIN Office dataset (collected by prior work)
Benchmarks: Franka Kitchen (MKBL, MLSH) CALVIN Office cleaning
Metrics: average completed subtasks (ACS) number of leaf nodes learning curves
Baselines: SPiRL VQ-SPiRL BC+SAC CART
Builds on: SPiRL VQ-VAE soft actor-critic VIPER
Closest papers by meaning
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.86)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.85)
- RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space (NeurIPS 2024 · similarity 0.85)
- Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization (ICLR 2025 · similarity 0.85)
- SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes (NeurIPS 2025 · similarity 0.85)
- ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.