Research map / Multi-agent systems and games / LLM agents
HiPER: Hierarchical Plan–Execute RL for Multi-Turn LLM Agents
Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming, Zhaoran Wang, Alfredo Garcia, Mingyi Hong
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiPER, a hierarchical plan-execute RL method for training multi-turn LLM agents on long-horizon tasks with sparse, delayed rewards.
Problem
Training LLM agents for multi-turn decision-making with sparse and delayed rewards in long-horizon tasks, where flat policies operating at a single time scale struggle.
Approach
Models the LLM agent as a hierarchical policy with a planning level and an execution level operating at different time scales, rather than a flat per-turn action policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch the base model prompt from ReAct to Plan-Execute without RL training (ablation) | Qwen2.5-1.5B-Instruct base · ALFWorld | overall success rate (%): 8.3 -> 2.9 | Hurt |
| switch PPO from ReAct to Plan-Execute prompting (ablation) | PPO (1.5B) · ALFWorld | overall success rate (%): +13.1% | Improved |
| switch GiGPO from ReAct to Plan-Execute prompting (ablation) | GiGPO (1.5B) · ALFWorld | overall success rate (%): +4.4% | Improved |
| add HiPER hierarchical advantage estimation on top of Plan-Execute GiGPO (ablation) | GiGPO with Plan-Execute · ALFWorld | overall success rate (%): +4.2% | Improved |
| replace GiGPO with HiPER | GiGPO (Qwen2.5-1.5B) · ALFWorld | overall success rate (%): 86.7 -> 95.3 | Improved |
| replace GiGPO with HiPER | GiGPO (Qwen2.5-1.5B) · WebShop | success rate (%): 67.4 -> 71.4 | Improved |
| replace GiGPO with HiPER | GiGPO (Qwen2.5-7B) · ALFWorld | overall success rate (%): 90.8 -> 97.4 | Improved |
| replace GiGPO with HiPER | GiGPO (Qwen2.5-7B) · WebShop | success rate (%): 75.2 -> 83.3 | Improved |
| replace flat PPO/GRPO with HiPER for sample efficiency | PPO/GRPO (Qwen2.5-7B) · ALFWorld validation | steps to reach ~80% success: ~140 steps -> about 50 steps (2.8x) | Improved |
| replace flat PPO/GRPO with HiPER for sample efficiency | PPO/GRPO (Qwen2.5-1.5B) · ALFWorld validation | training speedup: 2.5x speedup | Improved |
| add RL training (PPO) to the base model | Qwen2.5-1.5B-Instruct · ALFWorld | overall success rate (%): 8.3 -> 68.2 | Improved |
| add RL training (PPO) to the base model | Qwen2.5-1.5B-Instruct · WebShop | success rate (%): 5.5 -> 51.5 | Improved |
Closest papers by meaning
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.90)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.89)
- Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.88)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.88)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.88)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
