Research map / Reinforcement learning / Visual and representation RL
Hierarchical Programmatic Option Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HIPO, a hierarchical framework that retrieves diverse, compatible, effective programs (via modified CEM in a learned program embedding space) as options and trains a PPO high-level policy to sequence them for long-horizon Karel tasks.
Problem
Programmatic RL policies are human-readable but only handle short-horizon tasks and fail to generalize to longer horizons; long repetitive tasks need reusable skills.
Approach
Retrieve a set of effective, diverse, and compatible programs as options by extending the Cross-Entropy Method in a learned program embedding space with a diversity multiplier and a compatibility-aware sequence return, then learn a high-level PPO policy that selects which program to execute next.
Key results
- On KAREL and KAREL-HARD, HIPO outperforms LEAPS and HPRL on 8 of 10 tasks (e.g., 1.00 on STAIRCLIMBER vs 0.50 for LEAPS and 0.50 for HPRL).
- On KAREL-LONG, HIPO achieves the best mean return on SEESAW (0.53), UP-N-DOWN (0.76), FARMER (0.62), and INF-DOORKEY (0.66), while HC is best on INF-HARVESTER (0.88 vs HIPO 0.79).
- In inductive generalization to 2x-16x longer horizons, HIPO shows the smallest performance drop on FARMER and INF-DOORKEY, and its longest execution reached 48k environment steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add diversity and compatibility to program option retrieval (ablation) | CEM option retrieval for HIPO · KAREL-LONG tasks | mean return | Helped |
| Add compatibility to diversity-based option retrieval (ablation) | HIPO · SEESAW | mean return: 0.28 ± 0.23 -> 0.53 ± 0.10 | Helped |
| Add compatibility to diversity-based option retrieval (ablation) | HIPO · INF-DOORKEY | mean return: 0.36 ± 0.26 -> 0.66± 0.07 | Helped |
| Add the diversity multiplier to CEM program search (ablation) | CEM · KAREL and KAREL-HARD tasks | mean return | Helped |
| Learn a high-level policy instead of choosing options randomly (ablation) | HIPO · KAREL-LONG tasks | mean return | Helped |
| Replace programmatic options with primitive actions (ablation) | HIPO high-level policy · KAREL-LONG tasks | mean return | Hurt |
Surprising findings
- Hill climbing (HC) outperforms HIPO on INF-HARVESTER (0.88 vs 0.79).
- CEM+diversity underperforms plain CEM on several tasks in the single-program setting? No: CEM+diversity outperforms CEM on most tasks, but HIPO scores 0.36 on SNAKE, tying CEM+diversity and below DRL (0.67).
Limitations
- Evaluated only in the Karel grid-world domain with a custom DSL.
- Option set size |M| is fixed by hand (3 or 5) per task.
- HIPO does not dominate all baselines (e.g., HC better on INF-HARVESTER, DRL better on SNAKE and others in the short-horizon set).
Experimental setup
Datasets: KAREL problem set KAREL-HARD problem set KAREL-LONG problem set (new) randomly sampled Karel DSL programs
Benchmarks: KAREL KAREL-HARD KAREL-LONG
Metrics: mean return standard deviation over seeds program sample efficiency performance drop under extended horizon
Baselines: CEM LEAPS HPRL DRL (PPO) Option-Critic PAO HC (hill climbing) Random Transition
Builds on: LEAPS Option-Critic HPRL Cross-Entropy Method PPO
Closest papers by meaning
- Programmatic Reinforcement Learning without Oracles (ICLR 2022 · similarity 0.89)
- Learning to Synthesize Programs as Interpretable and Generalizable Policies (NeurIPS 2021 · similarity 0.87)
- Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs (ICML 2023 · similarity 0.87)
- Deep Reinforcement Learning from Hierarchical Preference Design (ICML 2025 · similarity 0.85)
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.85)
- Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.