Research map / Large language models / Agents, code and math
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TroVE, a training-free method where a code LM streams over examples, creating, importing, or skipping reusable Python functions, selects answers by execution agreement, and periodically trims the toolbox.
Problem
Program-aided LMs rely on primitive functions, producing verbose, error-prone programs; existing tool-making methods need training data, expert design, iteration, or produce large redundant toolboxes.
Approach
Maintain a shared, growing function library processed online; for each example sample responses in three modes (IMPORT, CREATE, SKIP), select by execution agreement plus solution complexity, and periodically remove rarely used functions with a logarithmic usage threshold.
Key results
- TroVE with CodeLLaMa-7b improves accuracy over Primitive on most of 11 datasets, e.g., TabMWP 0.43 to 0.47 and GQA 0.37 to 0.44, while using 79-98% smaller toolboxes than Instance.
- With GPT-4, TroVE reaches 0.72 on MATH algebra (vs CRAFT 0.68) and 0.92 on TabMWP (vs CRAFT 0.88) with libraries of 16 and 38 functions.
- In a human study on WTQ, verification was 31.4% faster than Primitive and detection accuracy rose 13% (0.77 to 0.87).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TroVE's reusable toolbox instead of primitive functions | CodeLLaMa-7B-Instruct · MATH algebra, count & probability, geometry, intermediate algebra, number theory, prealgebra, precalculus; TabMWP, WTQ, HiTab; GQA | accuracy: 0.15 -> 0.25; 0.14 -> 0.26; 0.06 -> 0.08; 0.05 -> 0.11; 0.16 -> 0.25; 0.21 -> 0.29; 0.10 -> 0.17; 0.43 -> 0.47; 0.20 -> | Helped |
| Use TroVE's shared, trimmed toolbox instead of per-example function induction | CodeLLaMa-7B-Instruct · MATH algebra, count & probability, geometry, intermediate algebra, number theory, prealgebra, precalculus; TabMWP, WTQ, HiTab; GQA | accuracy: 0.22 -> 0.25; 0.23 -> 0.26; 0.07 -> 0.08; 0.06 -> 0.11; 0.23 -> 0.25; 0.26 -> 0.29; 0.17 -> 0.17; 0.36 -> 0.47; 0.17 -> | Mixed |
| Remove periodic toolbox trimming (compare with keeping trimming) (ablation) | TroVE · MATH algebra | accuracy: 0.25 -> 0.25 | No clear effect |
| Remove periodic toolbox trimming (ablation) | TroVE · HiTab | accuracy: 0.18 -> 0.15 | Hurt |
| Remove periodic toolbox trimming (ablation) | TroVE · GQA | accuracy: 0.44 -> 0.39 | Hurt |
| Shuffle example order instead of using original dataset order (ablation) | TroVE · MATH algebra | accuracy: 0.25; 0.23--0.24 | No clear effect |
| Shuffle example order instead of using original dataset order (ablation) | TroVE · HiTab | accuracy: 0.18; 0.17--0.18 | No clear effect |
| Shuffle example order instead of using original dataset order (ablation) | TroVE · GQA | accuracy: 0.43; 0.43--0.44 | No clear effect |
Surprising findings
- Instance-level tool creation often underperforms the Primitive baseline on TableQA and VisualQA, generating hundreds to thousands of functions that confuse the model.
- A 7B CodeLLaMa2 with TroVE performs on par with GPT-4 on GQA (0.44 vs 0.44).
Limitations
- Results on GQA with primitives and some tasks depend on seen-primitive advantages, e.g., pandas familiarity.
- Reports the best of five runs to accommodate sampling randomness.
- GPT-4 comparisons are on only three datasets due to resource limits.
Experimental setup
Datasets: MATH TabMWP WTQ HiTab GQA
Benchmarks: MATH TabMWP WTQ HiTab GQA
Metrics: answer accuracy solution complexity (# ops) library size (# lib) human verification accuracy and time
Baselines: Primitive Instance LATM CRAFT CREATOR
Builds on: self-consistency VisProg LATM CREATOR CRAFT
Closest papers by meaning
- TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools (NeurIPS 2024 · similarity 0.86)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.86)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.86)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.85)
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.