Research map / Reinforcement learning / Visual and representation RL
OptionZero: Planning with Learned Options
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OptionZero, which adds an option network and option-aware dynamics to MuZero so options are discovered via self-play and used in MCTS, improving Atari human-normalized mean by 131.58%.
Problem
Planning with options in model-based RL typically relies on predefined options or expert demonstrations, and existing option planning recurrently evaluates every primitive step inside an option, which is costly when transitions are expensive.
Approach
Extend MuZero with an option head that predicts a dominant option (a sequence of primitive actions whose cumulative probability stays above 0.5) from each hidden state, and a dynamics network that can unroll whole options in one step, with option edges integrated into MCTS selection, expansion and backup.
Key results
- OptionZero with max option length 3 reaches 1054.30% mean human-normalized score vs 922.72% for MuZero (a 131.58% improvement).
- Max option length 6 reaches 1025.56% mean and 329.77% median, with 17 of 26 games improving over the baseline.
- In GridWorld, OptionZero reaches the shortest path (30 primitive actions) using only four learned options, about 7.5x faster training in that example.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned options with maximum length 3 | MuZero · 26 Atari games | mean human-normalized score: 922.72% -> 1054.30% | Helped |
| add learned options with maximum length 6 | MuZero · 26 Atari games | mean human-normalized score: 922.72% -> 1025.56% | Helped |
| increase maximum option length from 3 to 6 | OptionZero · 26 Atari games | mean human-normalized score: 1054.30% -> 1025.56% | Hurt |
| disable option execution in the environment and use options only for MCTS planning (ablation) | OptionZero ℓ3 · 26 Atari games | mean human-normalized score: 1054.30 % -> 1008.15 % | Hurt |
| disable option execution in the environment and use options only for MCTS planning (ablation) | OptionZero ℓ3 · Gopher | score: 68,240.60 -> 73,338.67 | Helped |
| disable option execution in the environment and use options only for MCTS planning (ablation) | OptionZero ℓ3 · Seaquest | score: 5,606.63 -> 3,378.60 | Hurt |
| use learned options | OptionZero ℓ3 · 26 Atari games | score | Mixed |
Surprising findings
- Only about 30% of search trees actually select an option, and the option-free ablation n-l3 still reaches 1008.15% mean, suggesting much of the gain comes from option-aware planning rather than execution.
- Performance drops in some games (e.g., bank heist, private eye) as option length increases, attributed to the dynamics network struggling with many option combinations.
Limitations
- Performance does not improve in all games, with some decreasing as option length grows, especially with numerous option types or visually complex observations.
- Requires a predefined maximum option length.
- Evaluated only on Atari and a toy GridWorld; not on continuous control or other domains.
Experimental setup
Datasets: Atari 2600 games (26 games) GridWorld (custom)
Benchmarks: Atari 2600 (26 games) GridWorld
Metrics: mean human-normalized score median human-normalized score average tree depth option usage proportions
Baselines: MuZero HIPS-epsilon Subgoal Search
Builds on: MuZero AlphaZero PUCT option-critic HIPS
Closest papers by meaning
- Planning in Stochastic Environments with a Learned Model (ICLR 2022 · similarity 0.88)
- On the role of planning in model-based deep reinforcement learning (ICLR 2021 · similarity 0.88)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.87)
- Flexible Option Learning (NeurIPS 2021 · similarity 0.87)
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.87)
- Strategic Planning: A Top-Down Approach to Option Generation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.