Research map / Multi-agent systems and games / LLM agents

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TASTE, which generates benchmark tasks by sampling valid tool sequences from an LLM-judged adaptive contrastive n-gram model, clustering them, and instantiating and evolving them into harder tasks, yielding τᶜ-Bench extending τ²-Bench.

Problem

Existing tool-use agent benchmarks such as τ²-Bench are saturating, and manual task construction is costly; manually written scenarios cover only a narrow subset of tool-use patterns.

Approach

Reverse the task construction pipeline: sample diverse valid tool sequences first from a contrastive n-gram model trained online with LLM plausibility labels, select representative medoids via K-medoids with a type-aware weighted Levenshtein distance, then instantiate and adversarially evolve tasks around them.

Key results

• Gemini-3-Flash scores 0.82–0.94 on τBV but 0.28–0.61 on τᶜ-Bench across domains and user simulators.
• Sampler validity rate rises from 6.7% (uniform sampling) to 86.7% with the full adaptive contrastive n-gram model.
• Generated tasks raise weighted edit distance by up to 124% and TTR by up to 111% relative to τBV.

Surprising findings

• Qwen-32B scores slightly higher on τᶜ-Bench than on τBV in Telecom with GPT-5.2 user (0.31 vs 0.28).
• Without training, the n-gram model initialized from τBV tasks reaches only 16.7% validity, so adaptive online training is essential.

Limitations

• Verifier agent has recall of 0.75 (Airline) and 0.83 (Retail), so it rejects some valid tasks.
• Evaluation restricted to final-state reward; trajectory-level metrics are excluded.
• Coverage and difficulty are defined through tool sequences and tuned on the τ²-Bench domains, and the approach is only demonstrated on conversational customer-support domains.

Experimental setup

Datasets: τ²-Bench Airline τ²-Bench Retail τ²-Bench Telecom τ-Bench Verified

Benchmarks: τᶜ-Bench τ-Bench Verified

Metrics: pass^1 pass^3 weighted edit distance type-token ratio tool-frequency entropy validity rate verifier precision verifier recall

Baselines: τ²-Bench Verified (τBV) uniform tool sampling

Builds on: τ²-Bench τ-Bench Verified Trajectory2Task ToolGrad K-medoids

Closest papers by meaning

• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.89)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
• PerfGuard: A Performance-Aware Agent for Visual Content Generation (ICLR 2026 · similarity 0.88)
• AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.88)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.88)