Research map / Large language models / Agents, code and math
Large Language Models as Tool Makers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LATM, a closed-loop framework where a strong LLM (GPT-4) creates reusable Python tools for task families and a cheaper LLM (GPT-3.5 Turbo) uses them, reducing serving cost while matching GPT-4 accuracy.
Problem
LLM tool use depends on availability of suitable tools, and running powerful models for every instance is expensive; there is no mechanism for LLMs to create reusable tools and amortize their cost across instances.
Approach
Split problem solving into a one-time tool-making stage by a powerful LLM (writes, verifies, and wraps a generic Python function from demonstrations) and a per-instance tool-using stage by a lightweight LLM that only generates function calls; add a dispatcher LLM that caches tools as a functional cache.
Key results
- With GPT-4 as tool maker, GPT-3.5 Turbo tool user reaches 79.7 on Logical Deduction (5) and 99.6 on Tracking Shuffled Objects (5) versus 66.4 and 61.6 for GPT-3.5 CoT.
- GPT-3.5 Turbo with LATM achieves 100.0 on Chinese Remainder Theorem and Schedule Meeting, where GPT-4 CoT gets 0.0 and 55.6.
- GPT-3.5 Turbo fails all 5 tool-making trials on Logical Deduction and Tracking Shuffled Objects, while GPT-4 succeeds in 3/5 and 4/5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Logical Deduction (5) | accuracy: 66.4 -> 79.7 | Helped |
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Tracking Shuffled Objects (5) | accuracy: 61.6 -> 99.6 | Helped |
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Dyck Language | accuracy: 20.4 -> 92.2 | Helped |
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Word Sorting | accuracy: 59.2 -> 98.3 | Helped |
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Chinese Remainder Theorem | accuracy: 0.0 -> 100.0 | Helped |
| Use a GPT-4-generated tool with GPT-3.5 Turbo instead of CoT | GPT-3.5 Turbo · Schedule Meeting | accuracy: 18.9 -> 100.0 | Helped |
| Use a GPT-4-generated tool with GPT-4 instead of CoT | GPT-4 · Logical Deduction (5) | accuracy: 88.8 -> 86.6 | Hurt |
| Use a GPT-4-generated tool with GPT-4 instead of CoT | GPT-4 · Dyck Language | accuracy: 63.6 -> 87.5 | Helped |
| Use GPT-3.5 Turbo instead of GPT-4 as tool maker (ablation) | LATM · Logical Deduction (5) | tool generation success: 3/5 -> 0/5 | Hurt |
| Use GPT-3.5 Turbo instead of GPT-4 as tool maker (ablation) | LATM · Tracking Shuffled Objects (5) | tool generation success: 4/5 -> 0/5 | Hurt |
| Use GPT-3.5 Turbo instead of GPT-4 as tool maker (ablation) | LATM · Word Sorting | tool generation success: 5/5 -> 5/5 | No clear effect |
| Use GPT-4-generated CoT instead of a reusable tool (ablation) | GPT-3.5 Turbo · Logical Deduction (5) | accuracy: 79.7 -> 36.8 | Hurt |
Surprising findings
- GPT-3.5 Turbo as tool user surpasses GPT-4 on Dyck Language (92.2 vs 87.5 for GPT-4 LATM), attributed to GPT-4 over-closing brackets when converting questions to function calls.
- Instruction-tuned GPT-3 models (text-davinci-002) underperform their pre-instruction-tuning counterparts as tool users on several tasks.
Limitations
- Tool-making fails for hard tasks with weaker tool makers, and tools may only work on training samples rather than generalize.
- Evaluated on only six tasks, with the dispatcher evaluated on a 100-sample mixed test set.
- Real-world safety, control, and tool quality validation are not addressed in depth.
Experimental setup
Datasets: BIG-Bench Logical Deduction (5) BIG-Bench Tracking Shuffled Objects (5) BIG-Bench Dyck Language BIG-Bench Word Sorting Chinese Remainder Theorem Schedule Meeting (constructed)
Benchmarks: BIG-Bench Logical Deduction (5) BIG-Bench Tracking Shuffled Objects (5) BIG-Bench Dyck Language BIG-Bench Word Sorting Chinese Remainder Theorem Schedule Meeting
Metrics: accuracy cost per n samples
Baselines: Chain-of-thought prompting GPT-4 CoT GPT-3.5 Turbo CoT GPTCache Chameleon
Builds on: Chain-of-thought prompting ReAct Self-debugging Programming by example
Closest papers by meaning
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.93)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.91)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.91)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.91)
- Gorilla: Large Language Model Connected with Massive APIs (NeurIPS 2024 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.