Research map / Large language models / Agents, code and math
Robust Function-Calling for On-Device Language Model via Function Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hammer, small on-device function-calling models fine-tuned on xLAM data plus 7.5k irrelevance examples and a function-masking scheme that replaces function/parameter names with random strings to improve cross-benchmark robustness.
Problem
Function-calling models show inconsistent performance across benchmarks, often because they over-rely on function and parameter naming conventions and fail to detect irrelevant functions.
Approach
Mask function and parameter names with random strings during fine-tuning so the model relies on descriptions, and augment training data with irrelevance examples where the correct function is removed and the target is an empty list.
Key results
- Hammer-7B reaches 83.92 overall accuracy on BFCL (as of 09/20/2024), ranking second behind GPT-4-0125-Preview among listed models and ahead of xLAM-7B-fc (79.41).
- Hammer-7B achieves an F1 average of 89.72 on function name and 76.21 on function plus arguments across academic benchmarks, above GPT-4-0613 (88.29 / 78.79 reported as name/args pair average) on the name metric.
- In the masking-name test on Seal-Tools, xLAM-1B-fc drops significantly while Hammer shows a much smaller drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2-7B-Instruct with function masking and irrelevance-augmented data (ablation) | Qwen2-7B-Instruct (Hammer-7B) · Academic benchmarks average | F1 Func. + Args: 59.84 -> 76.21 | Helped |
| Fine-tune Qwen1.5-4B-Chat with function masking and irrelevance-augmented data (ablation) | Qwen1.5-4B-Chat (Hammer-4B) · Academic benchmarks average | F1 Func. + Args: 42.25 -> 71.35 | Helped |
| Fine-tune Qwen2-1.5B-Instruct with function masking and irrelevance-augmented data (ablation) | Qwen2-1.5B-Instruct (Hammer-1.5B) · Academic benchmarks average | F1 Func. + Args: 52.67 -> 66.20 | Helped |
| Fine-tune Deepseek-Coder-7B-Instruct with the Hammer approach (ablation) | Deepseek-Coder-7B-Instruct · Academic benchmarks average | F1 Func. + Args: 45.67 -> 74.94 | Helped |
| Fine-tune Deepseek-Coder-1.3B-Instruct with the Hammer approach (ablation) | Deepseek-Coder-1.3B-Instruct · Academic benchmarks average | F1 Func. + Args: 19.78 -> 70.91 | Helped |
| Increase the function-masking ratio (ablation) | Qwen2-1.5B trained on Seal-Tools · Seal-Tools | performance (metric not specified) | Hurt |
| Increase the function-masking ratio (ablation) | Qwen2-1.5B trained on Seal-Tools · API-Bank | performance (metric not specified) | Helped |
| Add irrelevance-augmented training examples (ablation) | Qwen2-1.5B-Instruct · BFCL | irrelevance detection and function-calling performance | Mixed |
| Use approximately 10% irrelevance-augmented data (ablation) | Qwen2-1.5B-Instruct (Hammer) · BFCL | overall performance | Helped |
| Apply function masking during training to reduce reliance on function and parameter names | Hammer · Seal-Tools with names masked at test time | performance drop under name obfuscation | Helped |
| Use function masking for complex Parallel Multiple calls | Hammer-7B · BFCL Parallel Multiple | AST and executable accuracy: 84.08 AST; 85.00 Exec. | Helped |
Surprising findings
- Adding irrelevance data improves irrelevance detection but hurts function-calling accuracy, showing an inverse trade-off.
- Hammer-7B outperforms GPT-4 on executable function evaluation on BFCL.
Limitations
- Irrelevance-data proportion (about 10%) is tuned on one setup and may need adjustment for other models or datasets.
- Irrelevance detection trades off against function-calling accuracy.
- Evaluation relies on benchmarks where some results are reported with mixed metrics and a 100-example subset of Tool-Alpaca.
Experimental setup
Datasets: xLAM-function-calling-60k Seal-Tools training set BFCL API-Bank Tool-Alpaca Seal-Tools Nexus Raven API Evaluation
Benchmarks: BFCL API-Bank Tool-Alpaca Seal-Tools Nexus Raven API Evaluation
Metrics: accuracy AST accuracy executable accuracy F1 function name F1 function name and arguments
Baselines: GPT-4 GPT-4o xLAM-7B-fc xLAM-1.3B-fc Granite-20B-FunctionCalling Gorilla-OpenFunctions-v2-7B Functionary-Small-v3.1-8B Hermes-2-Pro
Builds on: xLAM-function-calling-60k DPO
Closest papers by meaning
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.88)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.88)
- APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (NeurIPS 2024 · similarity 0.88)
- Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.86)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.84)
- LLM Agents Already Know When to Call Tools - Even Without Reasoning (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.