Research map / Multi-agent systems and games / LLM agents

$\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces τ-Knowledge, an extension of τ-Bench with a new fintech domain τ-Banking where agents must retrieve from ~700 unstructured documents and coordinate tool calls to make policy-compliant database changes.

Problem

Existing agent benchmarks evaluate retrieval or tool use in isolation, not fully agentic, long-horizon conversational use of private unstructured knowledge bases coupled with state-changing tools.

Approach

A conversational benchmark where task success requires retrieving natural-language policy/product documents (via dense, BM25, terminal, or golden context) and invoking discoverable tools mentioned only in documentation, verified by final database state; evaluated with pass^k reliability.

Key results

• Best configuration GPT-5.2 (high) with terminal-use reaches 25.52 pass^1; Claude-4.5-Opus and Claude-4.5-Sonnet with terminal reach 24.74 and 22.42.
• Even with gold documents in context, Claude-4.5-Opus reaches only 39.69 pass^1 and pass^4 falls to 26.80.
• No-knowledge setup drops average pass^1 to about 2%, confirming tasks require retrieved information.

Surprising findings

• Even with gold documents provided in context, the best model reaches only 39.69 pass^1, showing reasoning over documents is a bottleneck beyond retrieval.
• Long-context setup with the full knowledge base peaks at about 12% pass^1, below targeted retrieval configurations.

Limitations

• User simulations are simplified and lack variation in expertise, colloquial language, and noisy input.
• Efficiency and latency metrics depend on current API serving characteristics.
• Evaluated only under unrestricted fully agentic search; few-shot search constraints not studied.

Experimental setup

Datasets: τ-Banking knowledge base (698 documents) τ-Banking tasks (97)

Benchmarks: τ-Banking

Metrics: pass^1 pass^k (k<=4) document recall average duration token usage

Baselines: text-embedding-3-large dense retrieval Qwen3-embedding-8B dense retrieval BM25 Golden retriever (gold docs in context) No-knowledge setup Long-context setup

Builds on: τ-Bench BM25 Terminal-Bench Terminus agent

Closest papers by meaning

• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (ICML 2026 · similarity 0.88)
• \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.87)
• T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning (NeurIPS 2025 · similarity 0.87)
• AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.87)
• Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.87)
• PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents (ICML 2026 · similarity 0.86)