Research map / Multi-agent systems and games / LLM agents

AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AppWorld-UL, a 516-task user-in-the-loop tool-use benchmark built by perturbing AppWorld tasks to require clarification, infeasibility reporting, or confirmation, with a knowledge-constrained LLM user simulator; frontier agents reach only 48.6% I-TGC.

Problem

Existing agent-user interaction benchmarks capture few interaction types, use unbalanced user simulators, and run in small environments with few state-changing APIs, so they do not test realistic user-in-the-loop tool use.

Approach

Perturb well-defined AppWorld tasks (removing info, making states infeasible, or adding high-cost consequences) so that the missing knowledge is an explicit QA set held by a simulated user; the user LLM answers only from this knowledge set, and interaction recall is checked programmatically via an extended unit test.

Key results

• Claude Opus 4.7 with the Code Agent achieves 48.6% I-TGC and 30.2% I-SGC overall, and 35.7% I-TGC on the compositional subset.
• GPT-5.5 Code Agent reaches 78.1% I-TGC with oracle knowledge versus 41.8% default and 12.8% TGC when the user is hidden.
• Manual review of 312 simulated user responses found only 4 incorrect (1.3% error rate).

Surprising findings

• Open-weight GLM 5.1 (38.9% I-TGC) is close to frontier closed models on the Code Agent, while the function-calling agent does far worse than the code agent.
• Even with oracle knowledge, agents score below original AppWorld (78.1% vs 85.3% TGC), attributed to added task difficulty and information structure.

Limitations

• Compositional and perturbation tasks were manually built by the authors, taking about 6 hours per perturbation, limiting scale.
• Recall-only interaction metric does not penalize unnecessary questions; precision is reported but not used as primary signal.
• Function-calling results for GLM 5.1 were pending due to API access issues.

Experimental setup

Datasets: AppWorld

Benchmarks: AppWorld-UL AppWorld

Metrics: I-TGC I-SGC TGC SGC interaction recall interaction precision interaction deviation

Baselines: Function Calling Agent Code Agent (ReAct) GPT-5.5 Qwen 3.7 Max GLM 5.1 Kimi K2.6 Deepseek v4 Pro

Builds on: AppWorld ReAct tau-bench BFCLv3 UserBench

Closest papers by meaning

• SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.90)
• OpenApps: Simulating Environment Variations to Measure UI Agent Reliability (ICLR 2026 · similarity 0.89)
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.89)
• VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications (ICLR 2026 · similarity 0.88)
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.88)
• FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.88)