Research map / Multi-agent systems and games / LLM agents

$\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces τ-Voice, a full-duplex voice-agent benchmark extending τ²-bench with realistic audio, simulated turn-taking, and verifiable task completion, finding large voice-text performance gaps for GPT-5-era audio-native models.

Problem

Existing voice agent evaluations test conversational dynamics or speech robustness in isolation, not task completion with tool calls and database state under full-duplex, realistic audio conditions.

Approach

Extends τ²-bench to voice with a tick-based full-duplex orchestrator, a controllable voice user simulator (personas, noise, telephony effects, LLM-driven interruptions and backchannels) decoupled from wall-clock time, and metrics for both pass@1 task success and turn-taking quality.

Key results

• GPT-5 (reasoning) reaches 85% pass@1 on text, while voice agents reach 31-51% under Clean and 26-38% under Realistic conditions.
• Google loses 17% of its Clean performance under Realistic conditions versus 24-28% for OpenAI and xAI.
• Qualitative analysis attributes 79% (Voice-Fragile) and 90% (Noise-Fragile) of failures to agent behavior.

Surprising findings

• Accent effects are highly provider-specific: xAI loses 18pp on Retail with accents while Google is nearly unaffected.
• OpenAI has the highest Clean Retail score (71%) but the largest relative Realistic degradation, suggesting higher baseline capability does not protect against speech complexity.

Limitations

• English only and TTS-generated speech rather than recorded speech; accent results indicative only.
• Simulator is more patient than real users, with perfect memory and instantaneous tool calls.
• Agent speech is fed as transcripts to the simulator, bypassing ASR on the agent side.

Experimental setup

Datasets: τ²-bench tasks (Retail, Airline, Telecom)

Benchmarks: τ-Voice

Metrics: pass@1 responsiveness latency interrupt rate selectivity

Baselines: GPT-5 GPT-4.1 GPT-5.2 gpt-realtime-1.5 gemini-live-2.5-flash-native-audio grok-voice-agent

Builds on: τ-bench τ²-bench Full-Duplex-Bench

Closest papers by meaning

• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (ICML 2026 · similarity 0.89)
• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.87)
• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.87)
• Implicit Intelligence - Evaluating Agents on What Users Don’t Say (ICML 2026 · similarity 0.86)
• Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.85)