Research map / Large language models / Agents, code and math
TaskBench: Benchmarking Large Language Models for Task Automation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TaskBench, a benchmark built from Tool Graphs with back-instruct data generation, plus TaskEval metrics that evaluate LLMs on task decomposition, tool selection, and parameter prediction.
Problem
Lack of systematic, standardized benchmarks for evaluating LLMs on task automation, i.e., decomposing user instructions into sub-tasks, selecting tools with dependencies, and predicting tool parameters.
Approach
Model tools as a Tool Graph with resource and temporal dependencies, sample sub-graphs (node, chain, DAG), and use back-instruct to generate user instructions and tool invocation graphs from them, with LLM and rule-based critics plus human verification; evaluate across three stages with TaskEval.
Key results
- GPT-4 achieves 81.54 overall node F1 and 54.70 overall edge F1 on Hugging Face tool selection, with 90.90 node F1 on Multimedia tools.
- Back-Instruct scores 3.85 overall human rating versus 2.61 for Self-Instruct on 50 samples.
- Human-ranking consistency is high, averaging Kendall's tau 0.89 and Spearman's rho 0.78 across the three domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include sampled tool-graph edges in Back-Instruct prompts (ablation) | Back-Instruct · Human-rated generated instructions | Naturalness: 3.44 -> 3.89 | Helped |
| include sampled tool-graph edges in Back-Instruct prompts (ablation) | Back-Instruct · Human-rated generated instructions | Complexity: 3.27 -> 4.01 | Helped |
| include sampled tool-graph edges in Back-Instruct prompts (ablation) | Back-Instruct · Human-rated generated instructions | Alignment: 3.62 -> 3.66 | No clear effect |
| include sampled tool-graph edges in Back-Instruct prompts (ablation) | Back-Instruct · Human-rated generated instructions | Overall: 3.44 -> 3.85 | Helped |
| increase demonstrations from 0-shot to 2-shot (ablation) | CodeLlama-13B · Daily Life APIs | n-F1: 71.55 -> 92.33 | Helped |
| increase demonstrations from 0-shot to 2-shot (ablation) | CodeLlama-13B · Daily Life APIs | v-F1: 44.21 -> 66.03 | Helped |
| increase tool-graph size from 1 to 8 nodes (ablation) | GPT-4 · Task automation graph prediction | graph accuracy: 96.16% -> 25.00% | Hurt |
Surprising findings
- Edge prediction is consistently harder than node prediction, with F1 gaps of roughly 20% across models.
- Graph accuracy for GPT-4 drops from 96.16% for single-tool tasks to 25% for 8-tool graphs.
Limitations
- About 12% of sampled GPT-4-generated samples contained errors in a human error analysis.
- Only three tool-graph domains are covered, and only resource and temporal dependencies are modeled.
- Human verification covered only 50 samples for quality ratings.
Experimental setup
Datasets: Hugging Face tools Multimedia tools Daily Life APIs
Benchmarks: TaskBench
Metrics: Rouge-1 Rouge-2 Rouge-L Node F1 Edge F1 Normalized Edit Distance parameter name F1 parameter name and value F1 Kendall's tau Spearman's rho
Baselines: APIBench ToolBench MetaTool API-Bank Self-Instruct
Builds on: Self-Instruct HuggingGPT
Closest papers by meaning
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR 2024 · similarity 0.91)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.90)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.90)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.90)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.89)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.