Research map / Reinforcement learning / Visual and representation RL
Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CHiRP, which invents symbolic abstract options from continually learned conditional abstraction trees (CATs) and plans over them with A* search for sample-efficient continual RL on long-horizon, sparse-reward tasks.
Problem
Autonomously discovering composable, reusable, and mutually independent options with abstract symbolic representations for long-horizon, sparse-reward, stochastic tasks in continual RL with unknown transition and reward functions.
Approach
Uses context-specific CATs (conditioned on context variables) and a context-specific distance to find state changes that serve as option endpoints, invents abstract options with symbolic initiation/termination sets, and plans over them with an A* search over a plannable CAT augmented with bridging option signatures.
Key results
- CHiRP consistently outperforms Option-Critic, CAT+RL, and PPO in fraction of tasks solved vs training steps across stochastic continual RL domains, averaged over 10 trials with 20-task streams.
- CAT+RL is reported as second-best; Option-Critic and PPO consistently underperform and fail to solve tasks within budget in all domains.
- In taxi, CHiRP invents four options (navigate to pickup, pickup, navigate to dropoff, dropoff) with minimal overlap in affected variables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CHiRP instead of CAT+RL | Continual RL across Maze World, Four Rooms World, Office World, Taxi World, and Minecraft · Maze World, Four Rooms World, Office World, Taxi World, and Minecraft | fraction of tasks solved within allocated timesteps | Helped |
| Use CHiRP instead of Option-Critic | Continual RL across Maze World, Four Rooms World, Office World, Taxi World, and Minecraft · Maze World, Four Rooms World, Office World, Taxi World, and Minecraft | fraction of tasks solved within allocated timesteps | Helped |
| Use CHiRP instead of PPO | Continual RL across Maze World, Four Rooms World, Office World, Taxi World, and Minecraft · Maze World, Four Rooms World, Office World, Taxi World, and Minecraft | fraction of tasks solved within allocated timesteps | Helped |
Surprising findings
- PPO and Option-Critic fail to solve tasks within the allotted budget in all domains despite being general-purpose baselines.
Limitations
- Assumes full observability and discrete actions.
- No optimality guarantees are provided (future work).
- Requires factored state variables and a stochastic simulator as input.
Experimental setup
Datasets: Maze World Four Rooms World Office World Taxi World Minecraft
Benchmarks: Maze World Four Rooms World Office World Taxi World Minecraft
Metrics: fraction of tasks solved within timestep budget sample efficiency
Baselines: Option-Critic CAT+RL PPO
Builds on: CAT+RL Option-Critic options framework single-outcome determinization
Closest papers by meaning
- Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.89)
- Reconciling Spatial and Temporal Abstractions for Goal Representation (ICLR 2024 · similarity 0.89)
- Hierarchical Planning and Learning for Robots in Stochastic Settings Using Zero-Shot Option Invention (AAAI 2024 · similarity 0.88)
- Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Learning Planning Abstractions from Language (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.