Research map / Large language models / Agents, code and math

MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models

ICLR 2025 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MTU-Bench, a tool-use benchmark (MTU-Instruct training data plus MTU-Eval GPT-free metrics) built by converting task-oriented dialogue datasets into multi-granularity tool-use scenarios, and fine-tunes LLaMA3-8B into MTU-LLaMA.

Problem

Existing tool-use benchmarks lack coverage of multi-turn, multi-tool and out-of-distribution scenarios, rely on synthetic instructions and GPT-based evaluation that is costly and unstable, and offer coarse metrics.

Approach

Automatically transform real task-oriented dialogue datasets (intents and slots) into ReAct-style tool-use data via GPT-4 synthesis, and evaluate with fine-grained rule-based metrics (SATS, TN, TO, TPR) that need no GPT or human judge.

Key results

• MTU-LLaMA reaches 92.31 TS and 81.63 TS in M-S/S-S-style single-tool settings, outperforming its base LLaMA3-8B-Instruct across all settings and metrics.
• On the OOD split, MTU-LLaMA averages 41.10 versus 25.16 for LLaMA3-8B; on API-Bank it reaches 45.66 avg, close to GPT-4's 44.45.
• Proposed metrics agree with human judgments (Pearson 0.83-0.88 metric-human vs 0.92-0.95 human-human).

Measured effects reported in the paper

Surprising findings

• Closed-source GPT-4 is not uniformly best; GLM-4-Plus and Qwen-Max outperform it on the hard set and in some multi-turn/multi-tool settings.
• ToolLLaMA, a tool-specific fine-tuned model, performs poorly across settings, suggesting limited generalizability.

Limitations

• Benchmark may not capture all real-world edge cases or highly complex interactions.
• Metrics are tied to predefined benchmarks and may not cover all future tool-use challenges.
• Training and synthetic tool-use data are generated by GPT-4, with only ~10% filtered by GPT-4 on the train split.

Experimental setup

Datasets: MultiWOZ SGD TaskMaster MetaLWOZ ATIS SNIPS

Benchmarks: MTU-Eval (normal and hard sets) MTU-Bench OOD split API-Bank ToolTalk

Metrics: tool selection accuracy parameter selection accuracy success rate averaged turn success rate soft averaged turn success rate task process rate tool number accuracy tool order accuracy

Baselines: GPT-4 GPT-3.5 Qwen-Max GLM-4-Plus DeepSeek V2.5 LLaMA3-70B Qwen2-72B ToolLLaMA

Builds on: ReAct ToolLLM InsTag

Closest papers by meaning

• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.90)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.89)
• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
• BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.88)