Research map / Large language models / Agents, code and math
The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BFCL, a large multi-category function-calling benchmark with AST-based deterministic evaluation, crowd-sourced real user queries, multi-turn and agentic (web search, memory, SQL) tracks, and evaluates a wide range of LLMs.
Problem
Existing function-calling benchmarks are narrow, rely on LLM-simulated users or LLM judges, or cannot scale because they require executing functions, so there is no standard reliable benchmark for LLM tool use.
Approach
Use AST substring matching as a deterministic, execution-free proxy for function-call correctness, combined with real user-contributed queries, stateful multi-turn API environments, and agentic memory/web/SQL tasks, all in one leaderboard.
Key results
- Top model o1-2024-12-17 (FC) reaches only 12% on the memory category, showing memory management remains an open challenge.
- gpt-4o-2024-11-20 (Prompt) achieves 66.4 overall accuracy, the top overall score in Table 1.
- AST-based scores show strong correlation with execution-based scores on the single-turn dataset, supporting AST as an offline proxy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use native function-calling mode instead of prompting mode | Models supporting both modes, on BFCL · Function-call decoding | decoding issues: 412.93 vs. 182.5 out of 4,251 total entries | Helped |
| Use native function-calling mode instead of prompting mode | Models supporting both modes, on BFCL multiple function-call category · Multiple function-call category | incorrect function-call counts: 77.5 vs. 21 | Hurt |
| Evaluate on crowd-sourced queries rather than single-turn queries | Salesforce xLAM-7B · Single-turn vs. crowd-sourced | perplexity: 3.67 to 5.09 | Hurt |
| Evaluate on crowd-sourced queries rather than single-turn queries | Salesforce xLAM-7B · Single-turn vs. crowd-sourced | Char-NLL: 0.340 to 0.427 | Hurt |
| Evaluate on crowd-sourced queries rather than single-turn queries | Llama2-7B · Single-turn vs. crowd-sourced | perplexity: 3.47 on single-turn compared to 2.56 on crowd-sourced | Helped |
| Evaluate on crowd-sourced queries rather than single-turn queries | Llama2-7B · Single-turn vs. crowd-sourced | Char-NLL: 0.344 versus 0.264 | Helped |
Surprising findings
- Parallel function-call ability appears to regress in flagship models such as o1-2024-12-17 (FC) and claude-3.5-sonnet-20241022 (FC) compared with earlier versions.
- Prompting-mode models often outperform their FC-mode counterparts despite having about three times more decoding issues.
Limitations
- Crowd-sourced and multi-turn categories rely on LLM-assisted or human-edited data, and the multi-turn set is relatively small (800 entries).
- Perplexity-based contamination analysis covers only a few open-source models and shows mixed signals (xLAM anomaly).
Experimental setup
Datasets: BFCL single-turn dataset BFCL crowd-sourced dataset (64,517 collected queries, 2,251 curated) BFCL multi-turn dataset BFCL agentic dataset (web search, memory, SQL)
Benchmarks: BFCL
Metrics: accuracy AST accuracy execution accuracy state-based accuracy response-based accuracy perplexity char-NLL
Baselines: Gorilla APIBench ToolBench ToolSandbox TauBench StableToolBench T-Eval Nexus Function Calling Leaderboard
Builds on: Gorilla APIBench Toolformer ToolBench
Closest papers by meaning
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.90)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.90)
- Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.89)
- Robust Function-Calling for On-Device Language Model via Function Masking (ICLR 2025 · similarity 0.88)
- APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (NeurIPS 2024 · similarity 0.88)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.