Research map / Multi-agent systems and games / LLM agents

The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Toolathlon, a benchmark of 108 diverse, realistic, long-horizon, execution-verified tasks across 32 apps and 604 tools, showing the best model (Claude-4.5-Sonnet) reaches only 38.6% pass@1.

Problem

Existing tool-using language agent benchmarks are narrow in domains, mostly single-application, use synthetic or empty initial states, and have simplified short-horizon tasks, so they do not measure real-world long-horizon multi-app execution.

Approach

Combines real or containerized applications (many via MCP servers) with realistic initial environment states, fuzzy user-style instructions, and deterministic per-task evaluation scripts that check final environment states, run in isolated parallel containers.

Key results

• Claude-4.5-Sonnet achieves the top pass@1 of 38.6% with 20.2 average turns on Toolathlon.
• DeepSeek-V3.2-Exp is the best open-weights model at 20.1% pass@1.
• Pass^3 for Claude-4.5-Sonnet is 20.4%, showing a large gap from pass@3 (51.9%), indicating inconsistency across runs.

Surprising findings

• Increased reasoning effort (GPT-5 vs. GPT-5-high) gave no benefit on agentic tasks.
• Claude-4.5-Sonnet and GPT-5 score higher on Hard tasks than Medium tasks, suggesting failures stem from premature task termination rather than long horizons alone.

Limitations

• Only 108 tasks, and the benchmark is evaluated primarily with API models rather than fine-tuned agents.
• Some remote environments cannot be fully reset, so local containerized substitutes (Poste.io, WooCommerce, Canvas) are used instead of the real services.
• Tasks take 4-6 hours per task to implement, limiting scale.

Experimental setup

Datasets: Google Sheets real financial spreadsheets Canvas course data Snowflake data Notion databases

Benchmarks: Toolathlon

Metrics: pass@1 pass@3 pass^3 average turns output tokens cost

Baselines: Claude-4-Sonnet GPT-5 GPT-5-high Grok-4 Claude-4.5-Haiku o3 o4-mini Gemini-2.5-Pro DeepSeek-V3.2-Exp GLM-4.6 Qwen-3-Coder Kimi-K2-0905

Builds on: Model Context Protocol OpenAI Agents SDK pass^k

Closest papers by meaning

• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.90)
• GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.89)
• \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.89)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.89)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.89)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.88)