Research map / Multi-agent systems and games / Cooperative multi-agent RL
Accelerating Task Generalisation with Multi-Level Skill Hierarchies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fracture Cluster Options (FraCOs), a bottom-up multi-level hierarchical RL method that clusters frequent action-sequence "fractures" from experienced trajectories and keeps the clusters with highest expected usefulness as options, improving transfer to unseen tasks.
Problem
Generalising reinforcement learning agents to new tasks (reward and state distribution shifts) by reusing temporally-extended skills, which existing option and multi-level HRL methods struggle to transfer, especially with pixel observations.
Approach
Form fractures (state plus fixed-length action sequences) from successful trajectories, cluster them with HDBSCAN, score clusters by an expected-usefulness metric (Bayesian appearance probability, relative frequency, and usage entropy), and convert top clusters into options that are fixed once created and selected by a standard RL algorithm; repeat to build multiple levels.
Key results
- In tabular grid-worlds, learning on unseen reward tasks is progressively accelerated as hierarchy depth increases.
- In deep Procgen experiments, FraCOs and FraCOs-SSR improve both IID and OOD returns over OC-PPO, PPO and PPG on average across eight environments (eight seeds per experiment).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the FraCO hierarchy depth | FraCOs · Four Rooms, Nine Rooms, and Ramesh Maze unseen reward tasks | episodic return | Helped |
| increase the FraCO hierarchy depth | FraCOs · MetaGrid unseen 14×14 and 21×21 domains | episodic return | Helped |
| use FraCOs instead of OC-PPO, PPO, or PPG | Procgen environments · Procgen IID and OOD tasks | IQM return | Helped |
| transfer a Shared State Representation after warm-up (ablation) | FraCOs · Procgen IID and OOD tasks | IQM return | Helped |
Limitations
- HDBSCAN struggles to predict cluster assignments for new data as environments grow more complex.
- Only discrete action spaces are studied.
- Factorial growth of cluster search limits chain length and hierarchy depth.
Experimental setup
Datasets: Procgen
Benchmarks: Procgen Four Rooms Nine Rooms Ramesh Maze MetaGrid
Metrics: episodic return interquartile mean min-max normalised return
Baselines: PPO OC-PPO PPG
Builds on: Option-Critic PPO Discovery of Deep Options HDBSCAN
Closest papers by meaning
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.85)
- Meta-Reinforcement Learning via Exploratory Task Clustering (AAAI 2024 · similarity 0.85)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.84)
- Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks (ICML 2025 · similarity 0.84)
- Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (ICLR 2026 · similarity 0.84)
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.