Research map / Multi-agent systems and games / LLM agents

VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VitaBench, a benchmark of 400 tasks over 66 tools across food delivery, in-store consumption and online travel, with simulated users and a rubric-based sliding-window LLM evaluator; top models reach only ~30% on cross-scenario tasks.

Problem

Existing LLM agent benchmarks do not jointly capture reasoning over extensive information, complex inter-tool dependencies, and dynamic multi-turn user interaction in real-world life-serving applications.

Approach

Models tool dependencies as a graph with pre/post-conditions instead of verbose domain policies, composes single-domain scenarios into cross-scenario tasks, and evaluates long trajectories with a rubric-based sliding-window evaluator that carries forward rubric satisfaction state.

Key results

• Claude-4.1-Opus (w/o thinking) and o3 (high) reach 21.8% and 30.0% Avg@4 on cross-scenario tasks, the best results in the leaderboard.
• Single-scenario results: GPT-5 (high) reaches 54.0% Avg@4 on delivery and o3 reaches 37.8% on OTA; in-store best is 56.8% (LongCat-Flash-Thinking).
• Rubric-based sliding-window evaluator reaches Cohen's kappa 0.828 against human judgments, versus below 0.07 without rubrics.

Surprising findings

• In-store domain has the largest database but is easier than delivery, so database scale does not predict difficulty.
• Pass^4 is near zero for most models, indicating severe run-to-run instability despite temperature 0.

Limitations

• Most tasks and data are in Chinese; English version still in preparation.
• Evaluator and user simulator rely on LLMs (claude-3.7-sonnet and gpt-4.1), introducing stochasticity.
• Some thinking/API models could not be evaluated due to API stability issues.

Experimental setup

Benchmarks: VitaBench

Metrics: Avg@4 Pass@4 Pass^4 Cohen's kappa

Baselines: tau-bench tau2-bench ToolTalk ToolSandbox DialogTool UserBench MINT IN3

Builds on: tau-bench ToolSandbox UserBench

Closest papers by meaning

• AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.88)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.88)
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.88)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.88)