Research map / Large language models / Agents, code and math
Benchmarking LLM Tool-Use in the Wild
Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, feng zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from task order 1 to task order 4 within a dialogue | Gemini-2.0-Thinking · WildToolBench | task accuracy (%): 78.13 -> 50.39 | Hurt |
| move from task order 1 to task order 4 within a dialogue | Qwen2.5-0.5B-Instruct · WildToolBench | task accuracy (%): 43.75 -> 25.39 | Hurt |
| scale Qwen2.5-Instruct from 0.5B to 72B | Qwen2.5-Instruct · WildToolBench | session accuracy (%): 0.78 -> 6.25 | Improved |
| scale Qwen2.5-Instruct from 3B to 7B | Qwen2.5-Instruct · WildToolBench | session accuracy (%): 4.30 -> 4.30 | No clear effect |
| enable thinking mode | Qwen3-14B · WildToolBench | session accuracy (%): 7.81 -> 9.38 | Improved |
| enable thinking mode | Qwen3-32B · WildToolBench | session accuracy (%): 8.59 -> 7.81 | Hurt |
| enable thinking mode | Doubao-1.5 · WildToolBench | session accuracy (%): 0.78 -> 6.64 | Improved |
| enable thinking mode | Doubao-1.6 · WildToolBench | session accuracy (%): 7.03 -> 3.13 | Hurt |
| use tool-specialized model instead of general model (xLAM-2-70B vs GLM-4.5) | tool-use LLMs · WildToolBench | optimal path rate, overall (%): 39.42 -> 19.92 | Hurt |
| tasks requiring clarification instead of casual chat | Grok-4 · WildToolBench | task accuracy (%): 66.02 -> 33.59 | Hurt |
| mixed sequential+parallel instead of parallel-only compositional tasks | Claude-4-Sonnet · WildToolBench | task accuracy (%): 54.78 -> 25.00 | Hurt |
| more instruction transitions in dialogue | LLMs overall · WildToolBench | task accuracy: drop as much as 30% | Hurt |
Closest papers by meaning
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.90)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.90)
- LLM Agents Already Know When to Call Tools - Even Without Reasoning (NeurIPS 2026 · similarity 0.89)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.89)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.89)
- From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
