Research map / Reinforcement learning / Visual and representation RL

Strategic Planning: A Top-Down Approach to Option Generation

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Hierarchical skills and priors

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Strategist, an LLM tree-search agent that generates natural-language strategy trees and converts them into shaped rewards for RL (PPO, EDE), tested on modified and original Crafter.

Problem

Conventional bottom-up RL struggles with sparse rewards and limited exploration budgets in real-world decision problems; existing hierarchical RL and LLM-assisted methods rely on ad hoc sub-goal design or single plans.

Approach

Formalize the Strategy Problem as finding a distribution over policies balancing specificity (low entropy) and value; use GPT-4o to build a top-down Strategy Tree of natural-language approaches and sub-goals, then distill LLM-scored progress toward each strategy into a reward shaping term annealed into the environment reward.

Key results

• On Crafter-Medium, Strategy Breed + PPO reached hunting in 8/8 runs at 0.59M steps versus 4/8 runs at 1.35M steps for vanilla PPO.
• Strategist+EDE with the Breed strategy significantly outperforms vanilla EDE on Crafter-Medium, with agents learning sapling-to-cow behavior that vanilla EDE struggles with.
• On original Crafter, Strategist+EDE variants Fight, Resource, and Craft significantly outperform vanilla EDE in cumulative achievements, while the Needs strategy performs comparably.

Measured effects reported in the paper

Surprising findings

• LLM-estimated feasibility and value are miscalibrated for some strategies; crafting-based plans performed below initial estimates while breeding exceeded them.
• Strategist-guided agents may revert to simpler behaviors aligned with environment reward once the shaping weight annealed to zero.

Limitations

• LLM feasibility and value proxies can be miscalibrated, leading to overexploration of unproductive branches.
• After the shaping weight decays to zero, agents can revert to simpler strategies not aligned with the intended strategy.
• Only one final strategy is committed to during training; evaluation is limited to Crafter variants.

Experimental setup

Datasets: Crafter

Benchmarks: Crafter Modified Crafter (Easy and Medium)

Metrics: episode reward steps to achieve hunting/breeding milestones feasibility value cumulative achievements relative steps per achievement

Baselines: PPO EDE DreamerV3 LLM-as-a-policy Eureka

Builds on: PPO EDE DreamerV3 Eureka

Closest papers by meaning

• StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction (NeurIPS 2026 · similarity 0.87)
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (ICLR 2026 · similarity 0.87)
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.86)
• Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.86)
• Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.86)
• Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.86)