Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes skill machines, FSMs built from reward machines of LTL tasks whose nodes compose pretrained goal-oriented skill primitives via Boolean operations, enabling zero-shot satisficing temporal-logic task solving with optional few-shot off-policy refinement.
Problem
Agents must solve many language/LTL-specified tasks combining spatial (Boolean) and temporal composition, but prior methods compose only one way, and the combinatorial space of tasks makes learning each task infeasible.
Approach
Learn skill primitives as goal-oriented world value functions for propositions and constraint literals, then build a skill machine from an RM by value-iterating over the RM and greedily selecting Boolean compositions (eventually-satisfy and always-satisfy constraints) per FSM state, with provable satisficing under reachability assumptions.
Key results
- In the Office Gridworld, skill machines achieve near-optimal zero-shot returns from the first epoch, and QL-SM (skill machine plus Q-learning) converges faster and higher than QL, CRM, HRM and reward-shaped baselines, including on the hardest Task 4.
- In Moving Targets with function approximation, zero-shot SM policies are very close to optimal on average, though compounding errors cause failures on Tasks 3 and 4.
- In Safety Gym, composed skill primitives achieve six increasingly complex temporal logic tasks zero-shot in a continuous domain, shown qualitatively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pair skill machines with Q-learning for few-shot task-specific learning | Skill Machines · Office Gridworld | average return | Helped |
| Initialize learning with skill-machine-composed skills | QL-SM · Office Gridworld | average return | Helped |
| Use zero-shot skill-machine composition instead of learning a task-specific policy | Skill Machines · Office Gridworld, Tasks 1–4 | average return | Mixed |
| Compose pretrained skill primitives to solve temporal-logic tasks without further learning | Skill Machines · Moving Targets | average return | Helped |
| Use zero-shot skill machines with sub-optimal primitives on challenging temporal constraints | Skill Machines · Moving Targets, Tasks 3 and 4 | task success | Hurt |
| Use few-shot Q-learning with skill-machine guidance when zero-shot behavior is sub-optimal | QL-SM · Office Gridworld with a broken global-reachability assumption | task performance | Helped |
| Use zero-shot skill-machine planning when the greedy transition expression is unsatisfiable | Skill Machines · Office Gridworld, task with an absorbing coffee location | task success | Hurt |
| Compose learned skill primitives for temporal-logic tasks in continuous control | Skill Machines · Safety Gym | task success | Helped |
Surprising findings
- Zero-shot skill machines can fail entirely when the global reachability assumption is violated, e.g. a coffee-office task where the agent cannot satisfy a required Boolean expression, although few-shot QL-SM still recovers the optimal policy.
Limitations
- Zero-shot policies are only recursively optimal and can be sub-optimal, with compounding errors in hard tasks like Tasks 3 and 4 in Moving Targets.
- Satisficing guarantees require global reachability of all Boolean expressions from any state; violations can cause zero-shot failure.
- Assumes deterministic dynamics and tabular or low-dimensional settings for most experiments.
Experimental setup
Datasets: Office Gridworld Moving Targets domain Safety Gym domain
Benchmarks: Office Gridworld Moving Targets Safety Gym
Metrics: average return zero-shot performance learning curves over time steps
Baselines: Q-learning counterfactual reward machines (CRM) hierarchical reward machines (HRM) reward-shaped QL-RS CRM-RS HRM-RS DQN optimal policies
Builds on: reward machines world value functions goal-oriented Q-learning hierarchical reward machines counterfactual reward machines Spot
Closest papers by meaning
- When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions (ICML 2024 · similarity 0.87)
- Zero-Shot Instruction Following in RL via Structured LTL Representations (NeurIPS 2026 · similarity 0.87)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.86)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.86)
- Lifting Traces to Logic: Programmatic Skill Induction with Neuro-Symbolic Learning for Long-Horizon Agentic Tasks (ICML 2026 · similarity 0.86)
- Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.