Research map / Multi-agent systems and games / LLM agents
Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Cognitive Load Theory-based framework (Tool Interaction Graph for intrinsic load, LLM-scored ambiguity/distraction for extraneous load) and the ToolLoad-Bench benchmark to map tool-use agents' capability boundaries via an exponential accuracy-vs-load model.
Problem
Final-accuracy benchmarks for LLM tool use obscure the cognitive bottlenecks and capability limits of agents as task complexity varies.
Approach
Decomposes task complexity into intrinsic load (a Tool Interaction Graph encoding dependency distance and selection interference) and extraneous load (LLM-rated query ambiguity and distractor tools), combines them additively with an empirically fitted weight, and models accuracy as exp(-(k·CL+b)) to yield per-model sensitivity and baseline parameters.
Key results
- xLAM2-32B achieves the highest overall accuracy on ToolLoad-Bench at 78.8%, versus 68.0% for GPT-4o.
- Fitted load sensitivity k is lowest for xLAM2-32B (0.034) with baseline b=1.22; Qwen3-8B has k=0.085 and b=1.12.
- Hosmer-Lemeshow tests give p-values from 0.10 to 0.89 across models, interpreted as good calibration.
Surprising findings
- Llama3.3-70B reaches only 17% overall accuracy (23% in low-load regime), far below smaller models such as Qwen3-32B.
Limitations
- Extraneous load relies on LLM (Gemini 2.5 Pro) scoring rather than objective metrics.
- Domain coverage is limited to the constructed tool categories.
- Calibration checks are on the same data used to fit the model, and the total load weight is derived from the observed accuracy drops.
Experimental setup
Datasets: Berkeley Function Calling Leaderboard v3 multi-turn base
Benchmarks: ToolLoad-Bench
Metrics: accuracy Hosmer-Lemeshow chi-square goodness-of-fit p-value fitted k and b of exponential decay
Baselines: GPT-4o GPT-4o-mini Claude 3.7 Sonnet Gemini 2.5 Pro Qwen3-235B Qwen3-32B Qwen3-8B Llama3.3-70B xLAM2-32B
Builds on: Berkeley Function Calling Leaderboard Cognitive Load Theory
Closest papers by meaning
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.88)
- What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? (ICML 2026 · similarity 0.87)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)
- A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks (NeurIPS 2026 · similarity 0.87)
- T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning (NeurIPS 2025 · similarity 0.87)
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.