Research map / Multi-agent systems and games / LLM agents

$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces τ²-bench, a dual-control benchmark (new Telecom domain modeled as Dec-POMDP) where both agent and simulated user have tools acting on a shared environment, with programmatic compositional task generation and ablations separating reasoning from communication.

Problem

Existing conversational agent benchmarks are single-control: only the agent can act with tools while the user passively provides information, unlike real technical support where users must take actions to change the shared world.

Approach

Extends the tau-bench paradigm to dual-control: the simulated user gets its own tools over a mocked device state, the environment constrains user behavior, and tasks are composed programmatically from atomic init/solution/assertion subtasks with verifiable success; ablations (solo, ground-truth-action modes) separate reasoning from coordination errors.

Key results

• GPT-4.1 pass^1 drops from 74% (Retail) and 56% (Airline) to 34% on the new Telecom domain.
• Claude 3.7 Sonnet reaches about 49% pass^1 on Telecom; o4-mini about 42%.
• Moving from no-user (solo) to dual-control mode drops pass^1 by 18% for GPT-4.1 and 25% for o4-mini; user simulator error rate is 16% on Telecom vs 40% on Retail.

Surprising findings

• Providing workflow-based policy hurts performance in the ground-truth action mode for both GPT-4.1 and o4-mini.
• Persona-free (None) tasks perform on par with or worse than Hard-persona tasks.

Limitations

• Does not model the expert-novice gap of real customer support users.
• Domain curation still relies heavily on human experts.
• User tool constraints were not applied to the existing Airline and Retail domains.

Experimental setup

Datasets: tau-bench retail tau-bench airline

Benchmarks: tau2-bench Telecom tau-bench Retail tau-bench Airline

Metrics: pass^1 pass^k user simulator error rate

Baselines: GPT-4.1-mini GPT-4.1 o4-mini Claude 3.7 Sonnet

Builds on: tau-bench Dec-POMDP IntellAgent ToolSandbox

Closest papers by meaning

• $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.89)
• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.89)
• $\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge (ICML 2026 · similarity 0.88)
• Implicit Intelligence - Evaluating Agents on What Users Don’t Say (ICML 2026 · similarity 0.88)
• AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.88)
• WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.88)