Research map / Large language models / Agents, code and math
Gorilla: Large Language Model Connected with Massive APIs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gorilla fine-tunes LLaMA-7B on self-instructed API-call data with retriever-aware training (RAT) and evaluates it on the new APIBench using AST sub-tree matching for accuracy and hallucination.
Problem
LLMs generate inaccurate or hallucinated API calls, especially across large, frequently updated sets of ML APIs with overlapping functionality and changing documentation.
Approach
Retriever-aware training appends retrieved (possibly wrong) API documentation to the prompt during fine-tuning while keeping the ground-truth API as target, teaching the model to judge retrieved docs and adapt to test-time documentation changes; AST sub-tree matching measures accuracy and hallucination.
Key results
- Gorilla zero-shot reaches 59.13% overall accuracy on TorchHub, versus 38.70% for GPT-4 and 48.38% for GPT-3.5, with 6.98% hallucination.
- On HuggingFace zero-shot, Gorilla reaches 71.68% accuracy versus 19.80% for GPT-4 and 16.81% for GPT-3.5.
- AST-based metric agrees with human evaluation (78% accuracy for both) on 100 sampled generations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an Oracle retriever with Gorilla instead of no retriever (ablation) | Gorilla · Torch Hub | accuracy: 59.13 -> 67.20 | Helped |
| Use an Oracle retriever with Gorilla instead of no retriever (ablation) | Gorilla · HuggingFace | accuracy: 71.68 -> 91.26 | Helped |
| Use an Oracle retriever with Gorilla instead of no retriever (ablation) | Gorilla · TensorFlow Hub | accuracy: 83.79 -> 94.16 | Helped |
| Use a BM25 retriever instead of no retriever (ablation) | Gorilla · Torch Hub | accuracy: 59.13 -> 37.63 | Hurt |
| Use a GPT-Index retriever instead of no retriever (ablation) | Gorilla · HuggingFace | accuracy: 71.68 -> 28.10 | Hurt |
| Use a GPT-Index retriever instead of no retriever (ablation) | Gorilla · TensorFlow Hub | accuracy: 83.79 -> 52.40 | Hurt |
| Add retriever-aware training with ground-truth API documents (ablation) | Gorilla · Torch Hub | accuracy: 59.13 -> 67.20 | Helped |
| Add retriever-aware training with ground-truth API documents (ablation) | Gorilla · HuggingFace | accuracy: 71.68 -> 91.26 | Helped |
| Use three-shot in-context examples instead of zero-shot prompting (ablation) | GPT-3.5 · Torch Hub | accuracy: 41.93 -> 73.11 | Helped |
| Use three-shot in-context examples instead of zero-shot prompting (ablation) | GPT-3.5 · Torch Hub | hallucination: 10.75 -> 72.58 | Hurt |
| Replace DocPrompting with Gorilla | HuggingFace API calls · HuggingFace | accuracy: 61.72 -> 71.68 | Helped |
| Replace DocPrompting with Gorilla | HuggingFace API calls · HuggingFace | hallucination: 17.36 -> 10.95 | Helped |
Surprising findings
- Augmenting LLMs with a naive retriever does not always improve performance and can sometimes hurt it.
- GPT-3.5 shows less API hallucination than GPT-4 across TorchHub, HuggingFace and TensorFlow Hub settings.
Limitations
- Evaluation is limited to single API calls and to ML hubs (TorchHub, TensorFlow Hub, HuggingFace), with HuggingFace evaluated only on domain-level correctness for baselines.
- AST matching does not verify execution; only 72% of generated code ran successfully in manual checks.
- Retrieval with current retrievers (BM25, GPT-Index) still shows large gaps to the oracle retriever.
Experimental setup
Datasets: APIBench HuggingFace model cards TorchHub model cards TensorFlow Hub model cards
Benchmarks: APIBench
Metrics: AST sub-tree matching accuracy hallucination rate error rate
Baselines: GPT-4 GPT-3.5-turbo Claude LLaMA-7B DocPrompting
Builds on: self-instruct LLaMA DocPrompting Toolformer
Closest papers by meaning
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.89)
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.89)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.88)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.87)
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.