Research map / Large language models / Agents, code and math
GTA: A Benchmark for General Tool Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GTA, a benchmark of 229 human-written, step-implicit real-world tool-use queries with multimodal image contexts and executable tool chains over 14 deployed tools, and evaluates 16 LLMs as agents.
Problem
Existing tool-use benchmarks rely on AI-generated queries, single-step tasks, virtual tools, and text-only inputs, failing to reflect real-world agent problem solving.
Approach
Human-designed queries with implicit tool use and multimodal image contexts, paired with executable tool chains over real deployed perception, operation, logic, and creativity tools, plus fine-grained step-level and end-to-end metrics.
Key results
- GPT-4 (GPT-4-1106-Preview) reaches 46.59% AnsAcc and GPT-4o 41.52%, while most other models score below 25%.
- ArgAcc is the lowest step-by-step metric for most models and has the highest correlation with final AnsAcc, indicating argument prediction is the main bottleneck.
- Agent-Flan-7B improves InstAcc from 30.86 to 71.60 over Llama-2-Chat-7B, but its AnsAcc only rises from 3.96 to 6.45.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-2-Chat-7B on ReAct and JSON instruction-following data | Llama-2-Chat-7B · GTA | AnsAcc: 3.96 -> 6.45 | Helped |
| Fine-tune Llama-2-Chat-7B on ReAct and JSON instruction-following data | Llama-2-Chat-7B · GTA | InstAcc: 30.86 -> 71.60 | Helped |
| Fine-tune Llama-2-Chat-7B on ReAct and JSON instruction-following data | Llama-2-Chat-7B · GTA | ToolAcc: 16.34 -> 41.11 | Helped |
| Fine-tune Llama-2-Chat-7B on ReAct and JSON instruction-following data | Llama-2-Chat-7B · GTA | ArgAcc: 0.36 -> 6.82 | Helped |
| Use GPT-4o instead of GPT-4-1106-Preview | GPT models · GTA | AnsAcc: 46.59 -> 41.52 | Hurt |
| Increase model size from 8B to 70B within the Llama-3 series | Llama-3 · GTA | AnsAcc: 3.1 -> 8.32 | Helped |
Surprising findings
- GPT-4 often fails by requesting more information or producing only thoughts without taking action, even when queries contain sufficient detail.
- Larger models do not consistently beat smaller models across different series (e.g., Llama-3-70B underperforms Qwen1.5-7B on AnsAcc).
Limitations
- All queries are in English, so language diversity is lacking.
- Human-written queries and tool chains make the benchmark costlier than AI-generated alternatives.
- Only 229 tasks, a relatively small test set.
Experimental setup
Datasets: GTA
Benchmarks: GTA
Metrics: InstAcc ToolAcc ArgAcc SummAcc AnsAcc AnsAcc w/ ImgGen tool-selection F1
Baselines: GPT-4-1106-Preview GPT-4o GPT-3.5-Turbo Claude-3-Opus Mistral-Large Qwen1.5-72B-Chat Llama-3-70B-Instruct Agent-Flan-7B
Builds on: ReAct Lagent AgentLego OpenCompass AgentFLAN
Closest papers by meaning
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.90)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.89)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.89)
- The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.89)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.88)
- Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.