Research map / Large language models / Agents, code and math

ToolACE: Winning the Points of LLM Function Calling

ICLR 2025 · Large language models › Agents, code and math · approach: Software engineering agents · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ToolACE, an agentic pipeline that synthesizes a 26,507-API pool and complexity-aware, rule- and model-verified function-calling dialogs, then fine-tunes LLaMA-3.1-8B to match GPT-4-class function calling.

Problem

Function-calling training data from real logs is hard to collect, and existing synthetic pipelines lack API coverage, dialog complexity, and accuracy, limiting LLM tool-use generalization.

Approach

A self-evolving API synthesis (speciation, adaptation, evolution) from pretraining documents, multi-agent dialog generation whose complexity is calibrated by the target model's own loss, and a dual-layer verifier combining rule checks with LLM-based checks.

Key results

• ToolACE-8B reaches 59.22 overall on the BFCL-v3 leaderboard (ranked 3rd), near GPT-4-turbo (59.49) and GPT-4o (59.29).
• On API-Bank Call, ToolACE-8B scores 75.94, matching gpt-4-0613 (75.94) and above LLaMA-3.1-8B-Instruct (71.18).
• Training on medium-complexity data (ToolACE_medium) gives slightly better BFCL results than easy or hard subsets.

Measured effects reported in the paper

Surprising findings

• Excluding multi-type (irrelevance) data drops irrelevance accuracy to 6.99%, showing the model otherwise rarely abstains from calling tools.
• Few-shot in-context learning with LLaMA-3.1-8B underperforms both zero-shot and fine-tuning on BFCL, with the model copying tools from the demonstrations.

Limitations

• Complexity evaluation cost scales with the size of the model being trained and the number of samples.
• Non-uniform complexity-based sampling may keep the model in its comfort zone; iterative multi-round sampling is left for future work.
• General capabilities (reasoning, understanding) still lag GPT-4 after function-calling fine-tuning.

Experimental setup

Datasets: ToolACE synthetic data ToolLLM data xLAM data pretraining documents (seed for API synthesis)

Benchmarks: BFCL-v3 API-Bank MMLU HumanEval GSM8K CommonSenseQA

Metrics: AST accuracy executable accuracy relevance irrelevance multi-turn accuracy overall accuracy API-Bank accuracy

Baselines: GPT-4-turbo GPT-4o GPT-4o-mini xLAM-7b-fc-r Gorilla-OpenFunctions-v2 Functionary LLaMA-3.1-8B-Instruct ToolLLM data xLAM data

Builds on: LoRA ToolLLM xLAM

Closest papers by meaning

• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.90)
• The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.90)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.90)
• Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.89)
• Learning Evolving Tools for Large Language Models (ICLR 2025 · similarity 0.89)
• AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.89)