Research map / Large language models / Agents, code and math
ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ShortcutsBench, a benchmark of 1414 real Apple Shortcuts APIs and 7627 human-annotated shortcuts to evaluate API-based LLM agents on API selection, parameter filling, and requesting user/system input.
Problem
Existing API-agent benchmarks use manually crafted or LLM-generated APIs and queries with limited complexity, and they fail to differentiate modern LLMs; they also ignore parameter filling and requests for missing input.
Approach
Build the benchmark from real Apple Shortcuts workflows: real APIs extracted from apps, human-written action sequences as golden labels, and queries generated by GPT-4o from natural-language shortcut descriptions, with evaluation of API selection, parameter filling, and ask-for-input recognition.
Key results
- Even 3B models achieve high accuracy (e.g., 88-93%) on existing benchmarks such as MetaTool and ToolBench, showing these benchmarks fail to differentiate LLM intelligence.
- Average API selection accuracy drops by 46% from difficulty (0,1] to (5,15], ranging from 38% (Gemini-1.5-Pro) to 58% (ChatGLM-4-Air).
- All agents perform poorly at recognizing need for user/system input, with overall accuracy between 30.55% (GPT-3.5-turbo) and 55.18% (DeepSeek-Coder-V2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase task difficulty from (0,1] to (1,5] actions | API-based agents on ShortcutsBench · API selection | accuracy: The average accuracy dropped by 19%; decreases ranged from 9% to 44%. | Hurt |
| Increase task difficulty from (0,1] to (5,15] actions | API-based agents on ShortcutsBench · API selection | accuracy: accuracy fell by 46%; drops ranged from 38% to 58%. | Hurt |
| Use open-source LLM agents instead of closed-source agents | API-based agents on ShortcutsBench · API selection, lower-difficulty tasks | accuracy | No clear effect |
| Use open-source LLM agents instead of closed-source agents | API-based agents on ShortcutsBench · API selection, highest-difficulty tasks | accuracy | Hurt |
| Fill parameters from the user's query rather than use outputs from previous actions | API-based agents on ShortcutsBench · API parameter value filling | accuracy: The accuracy drop ranges from 2.55% to 15.39%. | Hurt |
| Fine-tune AgentLM, Lemur, or xLAM models for agent scenarios | ShortcutsBench API agents · Recognition of need for input | accuracy | No clear effect |
| Increase task difficulty | Less-intelligent LLM agents on ShortcutsBench · API parameter value filling | accuracy | Hurt |
Surprising findings
- Existing benchmarks such as MetaTool and ToolBench give high accuracy even for 3B models, so they no longer separate LLMs by intelligence.
- Fine-tuned API-agent models (AgentLM, xLAM) did not improve recognition of need for input over their base models.
Limitations
- Queries generated by GPT-4o were validated only on a 100-sample human check (94/100 met criteria), and complex data types were excluded from queries.
- Context limits led to excluding shortcuts longer than 30 actions and those calling other shortcuts; API lists are capped at 20 per query and retrieval is not studied.
- Evaluation covers only Apple Shortcuts on macOS-style APIs, limiting generality beyond the Apple ecosystem.
Experimental setup
Datasets: Apple Shortcuts (7627 shortcuts from 14 sharing sites)
Benchmarks: ShortcutsBench
Metrics: API selection accuracy static parameter preset accuracy outputs from previous actions accuracy ask for necessary information accuracy
Baselines: MetaTool ToolLLM ToolBench ToolAlpaca APIBank ToolQA ToolLens AgentLM xLAM Lemur
Builds on: ToolLLM ReAct MetaTool ToolBench
Closest papers by meaning
- The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.90)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.89)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.89)
- Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.89)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.89)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.