Research map / Large language models / Agents, code and math
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CRAFT creates task-specific toolsets of reusable, verified code snippets from GPT-4 solutions and retrieves them at inference via multi-view matching, improving frozen GPT-3.5-Turbo on VQA, TabMWP, and MATH algebra without finetuning.
Problem
General-purpose tool collections lack task-specific APIs, and existing LLM tool creation methods produce unverified, non-reusable tools and use weak retrieval, limiting how LLMs can be customized to specialized tasks.
Approach
Create a large toolset offline by generating GPT-4 code solutions for sampled training problems, abstracting them into generic reusable functions, validating them on the original problems, and deduplicating; at inference, retrieve tools via multi-view matching over the problem, function name, and docstring, then add them to the prompt.
Key results
- On GQA, OK-VQA, and A-OKVQA with GPT-3.5-Turbo, CRAFT reaches 45.4, 33.4, and 30.8 SAcc versus 35.0, 15.4, and 15.6 for ViperGPT.
- On MATH algebra, CRAFT achieves 68.1 accuracy versus 65.0 for CREATOR and 58.2 for the vanilla tool-using method.
- With GPT-4 as backbone, CRAFT improves GQA SAcc from 51.4 (ViperGPT) to 55.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on GQA · GQA | SAcc: 37.9 -> 45.4 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on GQA · GQA | F1: 39.0 -> 48.8 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on OK-VQA · OK-VQA | SAcc: 18.4 -> 33.4 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on OK-VQA · OK-VQA | F1: 28.9 -> 43.0 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on A-OKVQA · A-OKVQA | SAcc: 17.8 -> 30.8 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on A-OKVQA · A-OKVQA | F1: 26.1 -> 40.6 | Helped |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on TabMWP · TabMWP | accuracy: 89.2 -> 88.4 | Hurt |
| Use CRAFT instead of the strongest listed baseline | GPT-3.5-Turbo on MATH algebra · MATH algebra | accuracy: 65.0 -> 68.1 | Helped |
| Add abstraction to the tool-creation pipeline (ablation) | CRAFT on GQA · GQA | SAcc: 37.1 -> 45.4 | Helped |
| Add abstraction to the tool-creation pipeline (ablation) | CRAFT on OK-VQA · OK-VQA | SAcc: 31.0 -> 33.4 | Helped |
| Add abstraction to the tool-creation pipeline (ablation) | CRAFT on A-OKVQA · A-OKVQA | SAcc: 28.0 -> 30.8 | Helped |
| Include problem matching in multi-view retrieval (ablation) | CRAFT on GQA · GQA | SAcc: 42.4 -> 45.4 | Helped |
Surprising findings
- Directly adding external Python library functions (NumPy, SciPy, etc.) fails to improve and can hurt math reasoning (41.1 vs 58.2 accuracy).
- LATM brings negative effects on all datasets, with TabMWP accuracy dropping to 9.3.
Limitations
- Only applicable to tasks solvable by writing code; pseudocode extension left to future work.
- Effectiveness depends heavily on the backbone's tool-using ability; open-source CodeLlama performed near random.
- Gains with GPT-4 backbone are smaller than with GPT-3.5-Turbo.
Experimental setup
Datasets: LLaVA instruction data COCO-2017 TabMWP training split MATH algebra training split GQA OK-VQA A-OKVQA
Benchmarks: GQA OK-VQA A-OKVQA TabMWP MATH
Metrics: SAcc F1 accuracy cyclomatic complexity
Baselines: CoT ViperGPT Program-of-Thoughts External library LATM CREATOR SimCSE retrieval BM25 retrieval
Builds on: ViperGPT Program-of-Thoughts LATM CREATOR chain-of-thought
Closest papers by meaning
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.93)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.91)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.90)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.90)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.90)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.