Research map / Large language models / Agents, code and math
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ToolLLM, a framework with the ToolBench instruction-tuning dataset (126k instances built with ChatGPT over 16k+ RapidAPI APIs), a DFS-based decision tree (DFSDT) for solution annotation, the ToolEval ChatGPT-based evaluator, and ToolLLaMA, a LLaMA-2 7B model fine-tuned for tool use with a neural API retriever.
Problem
Open-source LLMs lack tool-use capabilities for calling real-world APIs to fulfill complex, multi-tool instructions, because instruction tuning data largely ignores tool use and prior datasets have limited APIs, single-tool scenarios, and weak planning/reasoning.
Approach
Automatically build a large, real-API, multi-tool instruction dataset using ChatGPT function calls, annotate solution paths with a depth-first search decision tree (DFSDT) that backtracks from failed branches instead of single-path ReACT, and fine-tune LLaMA on the result, with an API retriever for large API pools.
Key results
- ToolLLaMA with DFSDT reaches an average pass rate of 66.7% across ToolBench test sets, close to ChatGPT+DFSDT (64.8%) and second only to GPT-4+DFSDT (71.1%).
- DFSDT improves average pass rate over ReACT for ChatGPT from 40.2% to 64.8%.
- The ToolEval ChatGPT evaluator agrees with humans at 87.1% on pass rate and 80.3% on win rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the API retriever instead of using BM25 | ToolBench API retrieval · Average across I1, I2, and I3 | NDCG@1: 18.5 -> 78.0 | Helped |
| Use DFSDT instead of ReACT@N | ChatGPT annotation · I1 | pass rate: 49.4 -> 58.0 | Helped |
| Use DFSDT instead of ReACT@N | ChatGPT annotation · I2 | pass rate: 49.4 -> 70.6 | Helped |
| Use DFSDT instead of ReACT@N | ChatGPT annotation · I3 | pass rate: 34.6 -> 62.8 | Helped |
| Use DFSDT instead of ReACT (ablation) | ToolLLaMA · ToolBench average | pass rate: 29.0 -> 66.7 | Helped |
| Use DFSDT instead of ReACT (ablation) | ToolLLaMA · ToolBench average | win rate: 47.0 -> 60.0 | Helped |
| Supply the top 5 retrieved APIs instead of the ground-truth API set (ablation) | ToolLLaMA + DFSDT · ToolBench average | pass rate: 66.7 -> 67.3 | Helped |
| Supply the top 5 retrieved APIs instead of the ground-truth API set (ablation) | ToolLLaMA + DFSDT · ToolBench average | win rate: 60.0 -> 63.1 | Helped |
| Use ToolLLaMA + our retriever instead of Gorilla-RS + BM25 | OOD API use · HuggingFace APIBench | AST accuracy: 15.71 -> 16.77 | Helped |
| Use ToolLLaMA + our retriever instead of Gorilla-RS + BM25 | OOD API use · TorchHub APIBench | AST accuracy: 50.00 -> 51.16 | Helped |
| Use ToolLLaMA + our retriever instead of Gorilla-RS + BM25 | OOD API use · TensorHub APIBench | AST accuracy: 41.90 -> 40.59 | Hurt |
Surprising findings
- Vicuna and Alpaca fail to pass any instruction (0% pass rate) despite extensive prompt engineering, indicating current instruction tuning does not cover tool use.
- Using retrieved top-5 APIs from the trained retriever slightly improves ToolLLaMA performance over the ground-truth API set (pass rate 67.3% vs 66.7% on average).
Limitations
- Relies on ChatGPT for data construction and evaluation, so annotation quality and evaluation depend on a closed-source model.
- Evaluation of tool use is intricate; human experts disagree on win-rate judgments, so the metric's reliability is limited.
- Gorilla cannot be evaluated on ToolBench's multi-tool and multi-step settings, limiting direct comparison.
Experimental setup
Datasets: ToolBench APIBench
Benchmarks: ToolBench APIBench
Metrics: pass rate win rate NDCG@1 NDCG@5 AST accuracy hallucination rate
Baselines: ChatGPT GPT-4 Text-Davinci-003 Claude-2 Vicuna Alpaca Gorilla BM25 text-embedding-ada-002 ReACT@N
Builds on: ReAct chain-of-thought Sentence-BERT positional interpolation AlpacaEval
Closest papers by meaning
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.91)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.91)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.90)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.90)
- Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.