Research map / Large language models / Agents, code and math
Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BUTTON, a bottom-up-then-top-down pipeline that synthesizes compositional multi-turn function-calling instruction data (BUTTONInstruct, 8k points) using GPT-4o, and fine-tunes Llama3 and Qwen2 models on it.
Problem
LLMs are mostly trained for single-turn function calling, lacking the ability to plan sequences of function calls for compositional, multi-step user queries; collecting such multi-turn function-calling data manually is costly.
Approach
Generate atomic tasks from scenarios, compose them into compositional tasks via sequential and parallel-then-sequential heuristics, synthesize functions guided by sub-task breakdowns, then simulate multi-turn trajectories with a multi-agent environment (user, assistant, tool agents).
Key results
- Llama3-8B answer accuracy on GTA improved from 1.4% to 30.5% after BUTTON tuning.
- Llama3-70B-BUTTON reaches 58.3% success rate on Tool-Query (All), versus 31.7% for Llama3-70B-Instruct and 41.7% for GPT-4-Turbo/Qwen2-72B-Instruct.
- Qwen2-72B-BUTTON achieves 45.7% answer accuracy on GTA, comparable to GPT-4o's 46.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama3-8B with BUTTONInstruct instead of general instruction tuning alone | Llama3-8B · GTA End-to-End | Ans.: 1.4 -> 30.5 | Helped |
| Fine-tune Qwen2-7B with BUTTONInstruct instead of general instruction tuning alone | Qwen2-7B · GTA End-to-End | Ans.: 13.1 -> 27.3 | Helped |
| Fine-tune Llama3-70B with BUTTONInstruct instead of general instruction tuning alone | Llama3-70B · GTA End-to-End | Ans.: 40.1 -> 43.5 | Helped |
| Fine-tune Qwen2-72B with BUTTONInstruct instead of general instruction tuning alone | Qwen2-72B · GTA End-to-End | Ans.: 27.3 -> 45.7 | Helped |
| Fine-tune Llama3-8B with BUTTONInstruct instead of general instruction tuning alone | Llama3-8B · Tool-Query All | Success Rate: 5.0 -> 35.0 | Helped |
| Fine-tune Qwen2-7B with BUTTONInstruct instead of general instruction tuning alone | Qwen2-7B · Tool-Query All | Success Rate: 23.3 -> 28.3 | Helped |
| Fine-tune Llama3-70B with BUTTONInstruct instead of general instruction tuning alone | Llama3-70B · Tool-Query All | Success Rate: 31.7 -> 58.3 | Helped |
| Fine-tune Qwen2-72B with BUTTONInstruct instead of general instruction tuning alone | Qwen2-72B · Tool-Query All | Success Rate: 41.7 -> 58.3 | Helped |
| Increase BUTTONInstruct training data from 2,000 to 8,000 examples (ablation) | Llama3-8B-BUTTON · Tool-Query All | Success Rate: 21.7 -> 35.0 | Helped |
| Increase BUTTONInstruct training data from 2,000 to 8,000 examples (ablation) | Llama3-8B-BUTTON · Tool-Query All | Process Rate: 56.5 -> 63.2 | Helped |
| Turn off parallel function calling by changing the system prompt (ablation) | Llama3-8B-BUTTON · Tool-Query All | Success Rate: 35.0% to 28.3% | Hurt |
| Turn off parallel function calling by changing the system prompt (ablation) | Llama3-8B-BUTTON · Tool-Query All | Process Rate: 63.2% to 58.7% | Hurt |
Surprising findings
- Tool-Query grounding accuracy is already high (~95%) for both original and tuned models, so the gap lies in planning rather than correct function use.
Limitations
- Data quality relies on prompts and procedures with no additional verification or filtering beyond the compositional-task consistency check.
- Functions are simulated by tool agents rather than actually implemented, so execution feedback is LLM-generated.
- Evaluation limited to two benchmarks (GTA, Tool-Query) and a single teacher model (GPT-4o).
Experimental setup
Datasets: glaive-function-calling-v2 ToolLLama dataset OpenHermes-2.5
Benchmarks: GTA Tool-Query
Metrics: Instruction accuracy Tool-selection accuracy Argument accuracy Summary accuracy F1 tool selection Final answer accuracy Grounding accuracy Process rate Success rate
Baselines: GPT-4o GPT-4-Turbo GPT-3.5-Turbo Llama3-8B-Instruct Llama3-70B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct
Builds on: ReAct Self-Instruct APIGen Chain-of-Instruct Sequential Instruction Tuning
Closest papers by meaning
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.89)
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
- Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition (ICML 2025 · similarity 0.87)
- Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models (AAAI 2025 · similarity 0.87)
- Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.