Research map / Large language models / Agents, code and math
Tool-Planner: Task Planning with Clusters across Multiple Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tool-Planner, which clusters tool APIs into toolkits via SimCSE embeddings and k-means, then plans over toolkits and falls back to alternative APIs within a toolkit before replanning across toolkits.
Problem
LLM tool-learning planners discard a whole planning path when one API call fails, and re-planning over many similar APIs is costly and unstable, especially for multi-tool tasks.
Approach
Group functionally similar APIs into clustered toolkits (SimCSE embeddings of LLM-generated API explanations, k-means++), make each plan node a toolkit rather than a single API, retry alternative APIs within the same toolkit on error, and replan across toolkits only when the whole toolkit fails.
Key results
- On ToolBench with GPT-4, Tool-Planner improves average pass rate by +8.8% and win rate by +9.1% over DFSDT (reported as 76.9 pass / 82.2 win on APIBench average vs DFSDT 70.3 / 67.7 on the same setting in tables).
- On APIBench with GPT-4, pass rate +6.6% and win rate +14.5% over DFSDT, with hallucination rate reduced by 2.1%.
- Tool-Planner latency is about 2x ReACT, while DFSDT is 6-8x higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use toolkit-based planning instead of DFSDT | GPT-4 on APIBench · APIBench Average | pass rate: +6.6% | Helped |
| use toolkit-based planning instead of DFSDT | GPT-4 on APIBench · APIBench Average | win rate: +14.5% | Helped |
| use toolkit-based planning instead of DFSDT | GPT-4 on APIBench · APIBench Average | hallucination rate: -2.1% | Helped |
| use toolkit-based planning instead of DFSDT | GPT-4 on ToolBench · ToolBench | pass rate: +8.8% | Helped |
| use toolkit-based planning instead of DFSDT | GPT-4 on ToolBench · ToolBench | win rate: +9.1% | Helped |
| add toolkit integration to ReACT (ablation) | ReACT with GPT-4 · ToolBench G1-Tool. | pass rate: 46.5 -> 55.5 | Helped |
| add toolkit integration to ReACT (ablation) | ReACT with GPT-4 · ToolBench G3-Inst. | win rate: 73.5 -> 77.8 | Helped |
| add toolkit integration to AdaPlanner (ablation) | AdaPlanner with GPT-4 · ToolBench G2-Inst. | win rate: 73.5 -> 76.0 | Helped |
| use SimCSE rather than RoBERTa-base embeddings for clustering (ablation) | Tool-Planner · ToolBench G1-inst. | pass rate: 60.5 -> 66.0 | Helped |
| use SimCSE rather than Contriever embeddings for clustering (ablation) | Tool-Planner · ToolBench G2-inst. | pass rate: 78.5 -> 83.5 | Helped |
| use SimCSE rather than text-embedding-ada-02 embeddings for clustering (ablation) | Tool-Planner · ToolBench G3-inst. | pass rate: 78.0 -> 83.0 | Helped |
| replace k-means++ with DBSCAN at ε=0.02 (ablation) | Tool-Planner · ToolBench G1-inst. | pass rate: 63.3 -> 64.3 | Helped |
Surprising findings
- Llama-2-13B as the sole model yields 0% pass and win for DFSDT, while Tool-Planner reaches roughly 10.9% pass with Llama-2-13B.
- Cluster-incomplete errors account for only 13.5% of failures, while invalid input parameters dominate the error distribution.
Limitations
- Performance depends heavily on clustering quality; large functional differences within a cluster can cause failure to find the right tool.
- Still roughly twice the latency of linear ReACT-style invocation.
- Results rely on GPT-4 as LLM judge for win rate and on small per-group test sets (100 instructions each).
Experimental setup
Datasets: ToolBench APIBench
Benchmarks: ToolBench APIBench
Metrics: pass rate win rate hallucination rate
Baselines: ReACT Reflexion AdaPlanner DFSDT ToolChain* Tree-Planner
Builds on: ReAct Reflexion AdaPlanner DFSDT ToolLLM
Closest papers by meaning
- ToolTree: Efficient LLM Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruning (ICLR 2026 · similarity 0.89)
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.88)
- Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.87)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.87)
- From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions (ICLR 2025 · similarity 0.87)
- Tree-Planner: Efficient Close-loop Task Planning with Large Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.