Research map / Large language models / Agents, code and math
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MetaTool, a benchmark with the ToolE dataset (~21k queries over 390 OpenAI plugin tools) to test whether LLMs know when to use tools and which tool to select, finding most eight evaluated LLMs perform poorly.
Problem
Existing tool-use benchmarks focus on tool invocation and execution, not on whether LLMs decide to use a tool or select the right tool(s) from candidate lists, which agents need; prior datasets also lack diversity and have overlapping tool functionality.
Approach
Builds a diverse query dataset from OpenAI plugin tool descriptions using multiple prompting styles (emotional, keyword, direct, detailed, multi-tool), resolves overlapping tools via merging/decomposition, and defines four tool-selection subtasks plus a tool-usage-awareness test with correct selection rate (CSR) as metric.
Key results
- Under zero-shot, only ChatGPT exceeds 70% on both accuracy and F1 for tool usage awareness (ChatGPT F1 77.81%; Llama2-13b F1 11.53%).
- For reliability-issue tool selection, most LLMs keep CSR below 20% even with 5-shot prompts (ChatGPT reaches 78.49% at 5-shot).
- In multi-tool selection, ChatGPT outperforms ChatGLM2 by nearly 70% in the multi-choice setting (2/2 CSR 88.28% vs 20.20%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use five-shot instead of zero-shot prompts | LLMs on tool usage awareness · Tool usage awareness | F1: Llama2-13b: 11.53 -> 54.32; +42.79% | Helped |
| Use five-shot instead of zero-shot prompts | Vicuna-7b · Tool selection with similar choices | CSR: 73.46 -> 63.67; -9.79 | Hurt |
| Use five-shot instead of zero-shot prompts | LLMs · Tool selection with possible reliability issues | CSR: +28.14 | Helped |
| Force selection of two tools rather than allowing zero, one, or two | Vicuna-33b · Multi-tool selection | CSR: 48.49 -> 91.15 | Helped |
| Increase the number of tools in the candidate list | LLMs · Tool selection in specific scenarios | CSR | Hurt |
| Use longer, more detailed tool descriptions | LLMs selecting tools · Tool selection | CSR | Helped |
| Rewrite tool descriptions with Llama2-70b | Llama2-13b · Tool selection | CSR: +7.83% | Helped |
| Rewrite tool descriptions with Llama2-70b | Vicuna series models · Tool selection | CSR | No clear effect |
| Rewrite tool descriptions with GPT-4 | ChatGLM and Llama2 series · Tool selection | CSR | Hurt |
| Rewrite tool descriptions with GPT-4 | Vicuna series · Tool selection | CSR | Helped |
Surprising findings
- Rewriting tool descriptions with GPT-4 sharply degrades ChatGLM and Llama2 performance while boosting Vicuna, so the best rewrite model depends on the downstream LLM.
- Vicuna-7b's few-shot prompting reduced its similar-tool selection CSR by about 10%.
Limitations
- Dataset generated by ChatGPT/GPT-4 and filtered by manual review, which may introduce generator bias.
- Tool set limited to 390 OpenAI plugin tools; multi-tool evaluation uses only 497 two-tool queries and pairs from the top 15 tools.
- Answer matching relies on rules plus manual analysis, which may be inconsistent for free-form responses.
Experimental setup
Datasets: ToolE (generated from OpenAI plugin descriptions) public daily-conversation and common-sense instruction datasets (negative samples) LIMA instructions
Benchmarks: MetaTool (ToolE-based tool usage awareness and tool selection tasks)
Metrics: accuracy precision recall F1 correct selection rate (CSR) Top@k error analysis
Baselines: ChatGPT ChatGLM2 Llama2-7b-chat Llama2-13b-chat Vicuna-7b Vicuna-13b Vicuna-33b Koala-13b
Builds on: ReAct ToolLLM APIBank
Closest papers by meaning
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.91)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.90)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.