Research map / Multi-agent systems and games / LLM agents

Non-Collaborative User Simulators for Tool Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a goal-aligned user simulator that exhibits four non-collaborative behaviors (unavailable services, tangential talk, impatience, incomplete utterances) and shows tool agents degrade substantially on MultiWOZ and τ-bench.

Problem

Existing user simulators for tool agents are agent-friendly and only cooperative, so agents are neither trained nor tested against non-collaborative users encountered in real deployments.

Approach

A user simulator built on a collaborative LLM simulator plus LLM modules (goal sharding with a dialogue state tracker, ending verifier, and behavior-specific generators) that injects four non-collaborative behaviors while guaranteeing all goal information is delivered.

Key results

• GPT-4.1-mini keeps most performance under single non-collaborative behaviors (e.g., 96.3% relative SR under unavailable service on MultiWOZ), while GPT-4.1-nano drops to 41.5% relative SR under tangential behavior on MultiWOZ.
• Tangential behavior causes the largest average degradation (average drop of 29.1%) among the four behaviors.
• The proposed simulator achieves about 70% overall win rate against PBUS in pairwise human evaluation with nine annotators.
• Fine-tuning Qwen2.5-3b-instruct on non-collaborative data raises average success from 78.8% to 86.9%.

Surprising findings

• Prompt-based simulators (PBUS) induce little performance degradation, whereas the proposed simulator with goal alignment induces substantial degradation.
• Apology behavior increases under impatient users, and this excessive apologizing hurts task completion under the step limit.

Limitations

• Goal alignment checker has false negatives (about 3% of 1,230 dialogues were marked misaligned and regenerated), with regeneration used to force alignment.
• Only a small number of agent models and two main benchmarks are evaluated; MultiWOZ agent environment is newly constructed.
• Behavior taxonomy is derived from marketing and user-LLM literature and may not cover all real-world non-collaborative users.

Experimental setup

Datasets: MultiWOZ tau-bench LMSYS-Chat-1M WildChat Persona Hub ColBench MINT

Benchmarks: MultiWOZ tau-bench ColBench MINT

Metrics: success rate relative success rate goal alignment initial goal alignment Matthews correlation coefficient

Baselines: Prompt-Based User Simulator (PBUS) collaborative user simulator

Builds on: tau-bench MultiWOZ ReAct Persona Hub

Closest papers by meaning

• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.87)
• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (ICML 2026 · similarity 0.87)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors (NeurIPS 2026 · similarity 0.86)
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction (ICLR 2026 · similarity 0.85)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.85)
• CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement (ICML 2026 · similarity 0.85)