Research map / Large language models / Agents, code and math
APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces APIGen, a multi-stage verified (format, execution, semantic) pipeline for generating diverse function-calling data, releasing a 60K-entry dataset and xLAM models that rank highly on BFCL.
Problem
Function-calling agents need diverse, reliable, high-quality training data, but existing agent datasets are static, rarely verified, and noisy, and rarely cover parallel function calls.
Approach
Generate query-answer pairs with LLMs from sampled executable APIs and seed examples, then verify each sample via format checking, real API execution, and an LLM semantic checker, with diversity from four query styles (simple, multiple, parallel, parallel multiple) and randomized sampling.
Key results
- xLAM-7B (FC) reaches 85.65% overall BFCL accuracy, ranking 6th and surpassing GPT-4o and Gemini-1.5-Pro in the leaderboard snapshot of 06/15/2024.
- xLAM-1B (FC) reaches 74.41% overall, ranking 24th and outperforming GPT-3.5-Turbo-0125 and Claude-3-Haiku.
- Human evaluation on 600 samples found about 95.3% of data of very high quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DeepSeek-Coder-7B-instruct-v1.5 on APIGen-generated data | DeepSeek-Coder-7B-instruct-v1.5 · BFCL | overall accuracy | Helped |
| Fine-tune DeepSeek-Coder-1.3B-instruct on APIGen-generated data | DeepSeek-Coder-1.3B-instruct · BFCL | overall accuracy | Helped |
| Add data filtered by the semantic checker to the training set (ablation) | xLAM models trained on APIGen data · BFCL | overall accuracy | Hurt |
| Add data filtered by the execution checker to the training set (ablation) | xLAM models trained on APIGen data · BFCL | overall accuracy | Hurt |
Surprising findings
- Adding data that failed the execution or semantic checks to training harms BFCL performance, with a larger impact on the smaller model.
- Weaker generators (DeepSeek-Coder-33B, Mixtral-8x7B) had pass rates of only 34% and 38%, versus 84% for DeepSeek-V2-Chat.
Limitations
- Only REST APIs and Python functions are supported.
- Only single-turn function calling is implemented.
- Model weights are not released pending legal approval.
Experimental setup
Datasets: ToolBench Berkeley Function-Calling Benchmark seed styles
Benchmarks: Berkeley Function-Calling Leaderboard (BFCL)
Metrics: overall accuracy AST accuracy executable function accuracy relevance detection accuracy
Baselines: GPT-4o GPT-4-Turbo Gemini-1.5-Pro Claude-3-Haiku GPT-3.5-Turbo Gorilla OpenFunctions-v2 Command-R-Plus Llama-3-70B-Instruct
Builds on: ToolBench AgentOhana xLAM
Closest papers by meaning
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.89)
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.88)
- Robust Function-Calling for On-Device Language Model via Function Masking (ICLR 2025 · similarity 0.88)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.86)
- API Pack: A Massive Multi-Programming Language Dataset for API Call Generation (ICLR 2025 · similarity 0.86)
- APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.