Research map / Large language models / Instruction and fine-tuning
API Pack: A Massive Multi-Programming Language Dataset for API Call Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces API Pack, a 1.1M-instance multi-language instruction dataset of API call code, and shows fine-tuned CodeLlama-13B beats GPT-3.5/GPT-4 on unseen APIs.
Problem
LLMs struggle to generate correct HTTP API call code, especially for unseen APIs and across programming languages, and no large open dataset existed for API call code generation.
Approach
Build a large multi-language instruction-API call dataset from OpenAPI specs (rendered into 10 languages via openapi-snippet), with LLM-generated and LLM-filtered instructions, and evaluate generalization across three levels of API novelty.
Key results
- Fine-tuned CodeLlama-13b reaches 49.5% API call accuracy on Level 3 (3-shot retrieved) vs 44.3% for GPT-4 and 39.5% for GPT-3.5.
- Retrieved 3-shot examples outperform random 3-shot examples for fine-tuned Mistral-7b and CodeLlama-13b across all levels.
- Mixing 100k cURL with 1k each of nine other languages yields cross-language performance comparable to expert per-language models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CodeLlama-13B on 20,000 Python API Pack instances with the 3-shot template instead of the 0-shot template (ablation) | CodeLlama-13B · Level 3 (unseen APIs and endpoints), 3-shot retrieved evaluation | API Call accuracy: 44.1% to 49.5% | Helped |
| Fine-tune CodeLlama-13B on 20,000 Python API Pack instances | CodeLlama-13B · Level 3 (unseen APIs and endpoints), 3-shot retrieved evaluation | API Call accuracy: 49.5% compared to 39.5% for GPT-3.5 and 44.3% for GPT-4 | Helped |
| Use retrieved relevant examples instead of randomly selected examples | Fine-tuned Mistral-7B and CodeLlama-13B · Levels 1–3 | API Call accuracy: 52.9 to 55.5 (Level 1); 49.2 to 51.4 (Level 2); 49.3 to 49.5 (Level 3) | Helped |
| Add generic reranking to retrieved examples (ablation) | Fine-tuned CodeLlama-13B · Level 1 (seen APIs and endpoints) | API Call accuracy: 55.5% to 52.9% | Hurt |
| Replace one retrieved example with the oracle example (ablation) | Fine-tuned CodeLlama-13B · Levels 1–3 | API Call accuracy: 71.2; 64.1; 59.3 | Helped |
| Combine 100,000 cURL instances with 1,000 instances in each of nine additional languages | Cross-language mixture model · API call generation across ten programming languages | API call performance | Helped |
| Increase API Pack fine-tuning data from 0 to 1 million instances | CodeLlama-13B · Levels 1–3, 0-shot evaluation | Endpoint and API call accuracy | Helped |
| Increase fine-tuning data toward 1 million instances | CodeLlama-13B · Levels 2 and 3, 3-shot evaluation | Endpoint and API call accuracy | Mixed |
| Train on APIs from four sources instead of ToolBench's single source | CodeLlama-13B · Levels 1–3, 0-shot and 3-shot | Endpoint and API Call accuracy: 37.8 to 55.5 (Level 1 API Call, 3-shot); 36.4 to 51.4 (Level 2 API Call, 3-shot); 38.1 to 49.1 (Level 3 API Call, 3-shot | Helped |
| Replace 45% of filtered instructions with non-filtered instructions (ablation) | API Pack fine-tuning · Levels 1–3, 0-shot and 3-shot | Endpoint and API Call accuracy: 54.9 to 55.5 (Level 1 API Call, 3-shot); 48.7 to 51.4 (Level 2 API Call, 3-shot); 49.5 to 49.1 (Level 3 API Call, 3-shot | Hurt |
Surprising findings
- Generic re-ranking (bge-reranker-large) does not consistently improve over plain retrieval and sometimes lowers accuracy.
- Filtering the LLM-generated instructions has only a limited effect: non-filtered data causes only a slight performance drop.
Limitations
- Evaluation relies on string similarity (difflib) rather than execution-based metrics such as pass rate.
- Dataset is single-API only and requires explicit API name, so multi-API scenarios are not covered.
- License information is limited for many APIs.
Experimental setup
Datasets: API Pack ToolBench Magicoder
Benchmarks: API Pack test split (Level 1, Level 2, Level 3) HumanEval+ MBPP
Metrics: API call accuracy endpoint accuracy intent accuracy difflib SequenceMatcher similarity >= 0.9
Baselines: GPT-3.5 GPT-4 Mistral-7b Llama-2-13b ToolBench-trained CodeLlama-13b Magicoder
Builds on: Self-Instruct Evol-Instruct Gorilla ToolBench
Closest papers by meaning
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.87)
- Gorilla: Large Language Model Connected with Massive APIs (NeurIPS 2024 · similarity 0.87)
- APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (NeurIPS 2024 · similarity 0.86)
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.86)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.85)
- McEval: Massively Multilingual Code Evaluation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.