Research map / Reinforcement learning / Visual and representation RL
Partially Observable Hierarchical Reinforcement Learning with AI Planning (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HFond-Plan, which encodes uncertainty about unknown predicates in a FOND planning model to derive a high-level plan that decomposes a POMDP into a tree of semi-POMDP options for hierarchical RL, evaluated on a MiniGrid key-door task against PPO and A2C.
Problem
Exploration in goal-oriented POMDPs with unknown predicates is inefficient because agents do not know how and when to stop exploring; fully exploring wastes samples on irrelevant variables.
Approach
Extends the PaRL framework to partial observability by modeling the agent's knowledge of unknown predicates with discovery effects in a FOND planning model, then deriving operator options (initiation, termination on prevail/discovery outcomes) for hierarchical RL.
Key results
- HFond-Plan's success rate over training samples is shown against PPO and A2C on a MiniGrid instance with unknown key locations and colors (Figure 2b), with the authors reporting guided exploration may improve performance.
- Evaluation used 4000 seeds per run (3900 train, 100 eval) and ten runs with different starting seeds per agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a discovery model using FOND planning to guide HRL exploration | HFondPlan · MiniGrid instance | success rate | Helped |
Limitations
- Preliminary evaluation on a single MiniGrid instance
- Replanning used to handle empty discovery effects; alternative approaches left to future work
- Assumes deterministic observation model for compilation, which may be violated
Experimental setup
Datasets: MiniGrid environment (procedurally generated seeds)
Benchmarks: MiniGrid key-door navigation instance
Metrics: success rate over training samples
Baselines: PPO A2C
Builds on: PaRL (Planning Annotated Reinforcement Learning) PPO A2C options framework
Closest papers by meaning
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.91)
- Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.88)
- Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach (AAAI 2025 · similarity 0.87)
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.87)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.87)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.