Research map / Large language models / Agents, code and math
Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, Estevam Hruschka
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restate all known variable values in each function return (ablation) | GPT-5 · FuncBenchGen (5 core nodes) | success rate: 62.5% -> 81.3% | Improved |
| restate all known variable values in each function return (ablation) | Gemini-2.5-Flash and Qwen3 · FuncBenchGen (5 core nodes) | success rate | No clear effect |
| increase core function calls from 5 to 20 (ablation) | GPT-5 · FuncBenchGen | success rate: 72.5% -> 15.0% | Hurt |
| increase core function calls from 5 to 20 (ablation) | Gemini-2.5-Pro · FuncBenchGen | success rate: 46.5% -> 6.0% | Hurt |
| add connected irrelevant functions (ablation) | all models · FuncBenchGen | success rate | Hurt |
| add disconnected irrelevant functions (ablation) | GPT-5 · FuncBenchGen | success rate | Improved |
| add disconnected irrelevant functions (ablation) | Gemini-2.5-Pro · FuncBenchGen | success rate | Hurt |
| increase dependency depth from 1 to 4-8 (ablation) | GPT-5 · FuncBenchGen (10 core nodes) | success rate: ~90% -> <30% | Hurt |
| use medium instead of minimal thinking budget (ablation) | GPT-5 · FuncBenchGen (5 core nodes) | success rate: <20% -> higher | Improved |
| increase core nodes to 40 (ablation) | GPT-5 · FuncBenchGen (disconnected) | success rate: <10% at 40 core nodes | Hurt |
| use reasoning-optimized model instead of general model | GPT-5 vs GPT-4.1 · FuncBenchGen (5 core nodes) | success rate: 12.0% -> 72.5% | Improved |
| path-like deeper graphs (depth 8-9) vs depth 5-7 (ablation) | GPT-5, Gemini-2.5-Pro, Qwen3 · FuncBenchGen (10 core nodes) | success rate | Mixed |
Closest papers by meaning
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.92)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.92)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
- Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.89)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.89)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
