Research map / Reinforcement learning / Visual and representation RL

PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planning

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: RL with tree search for discovery · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PhGPO, which distills successful tool-use trajectories into an ACO-style pheromone transition prior over an MCP tool-transition graph and uses it to guide GRPO rollouts for long-horizon tool planning.

Problem

Long-horizon multi-step tool planning suffers from combinatorial exploration; successful tool-transition patterns are absorbed implicitly into policy parameters and rarely reused when verified successes are rare.

Approach

Maintains an explicit, decaying pheromone value on tool-transition and tool-to-argument edges, deposited by verified successful trajectories and fused with task-dependent memories, then injects it as a log-space prior into top-K next-tool sampling during GRPO rollouts, with a progressive teacher-forcing curriculum.

Key results

• On Toolathlon with Qwen2.5-7B, PhGPO reaches 25.25% Match Ratio and 20.34% TSR, versus 21.26% / 18.91% for the strongest baseline TreeRL.
• On TOUCAN with Qwen2.5-7B, PhGPO reaches 35.17% Match Ratio and 26.28% TSR, above GTool (27.56% / 19.35%).
• Ablation on Toolathlon: removing the mixed curriculum drops TSR from 20.34% to 14.86%, the largest drop among the ablated components.

Surprising findings

• Freezing pheromones after the progressive pipeline (Static Prior) yields only a small drop (TSR 18.71% vs 20.34%), suggesting online updates matter less than the curriculum.

Limitations

• Evaluation relies on GPT-4o as an outcome judge for TSR, and training uses a simulator rather than real tool execution.
• Pheromone guidance depends on reference trajectories for deposition scores and teacher forcing, limiting applicability where references are unavailable.
• Argument-invocation modeling is simplified to frequent invocation patterns; the conclusion lists broader verification and evolving APIs as future work.

Experimental setup

Datasets: Toolathlon TRAJECT-Bench TOUCAN

Benchmarks: Toolathlon TRAJECT-Bench TOUCAN

Metrics: Match Ratio Task Success Rate

Baselines: ReAct Plan-and-Solve Beyond ReAct Gorilla ToolReAGt Tool Graph Retriever ToRL StepTool TreeRL ToolNet GAP NaviAgent

Builds on: GRPO ant colony optimization Model Context Protocol

Closest papers by meaning

• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• ToolTree: Efficient LLM Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruning (ICLR 2026 · similarity 0.87)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.86)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.86)
• ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search (ICLR 2024 · similarity 0.86)
• PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs (ICML 2026 · similarity 0.86)