Research map / Multi-agent systems and games / LLM agents

AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentNoiseBench, a framework that injects user-side and tool-side noise (via an adversarially optimized generator) into tau2-Bench, VitaBench and multi-hop QA, and evaluates 24 LLMs for robustness, finding that nearly all models degrade and tool noise hurts more.

Problem

Agent benchmarks assume idealized, clean user instructions and stable tool outputs, so measured performance overstates real-world robustness of tool-using LLM agents.

Approach

A taxonomy of user noise (ambiguity, conflict, redundancy, topic shift, boundary probing) and tool noise (execution failure, incomplete, erroneous, misleading, redundant), an automated injection pipeline that optimizes a noise-generation system prompt against a reference agent under a solvability constraint, and a trajectory-aware stability-gated success metric.

Key results

• Nearly all evaluated models degrade under noise, with an average accuracy drop of 20.8% (VitaBench 24.1%, tau2-Bench 21.2%, Search 16.8%).
• Thinking models show lower robustness than non-thinking models in most scenarios; e.g., gpt-5.2 (w/ thinking) reaches 0.81 clean vs 0.61 under tool noise on tau2-Bench.
• Tool noise hurts more than user noise across families; execution failures are the most destructive tool noise, reducing average score to 0.31.

Surprising findings

• Stronger reasoning does not confer noise robustness: reasoning models tend to rationalize injected noise into spurious reasoning chains.
• Agents are most sensitive to noise injected at the middle of the trajectory.

Limitations

• Evaluation targets language-centric tool-augmented agents and may not generalize to agents with explicit planning or asynchronous tools.
• Trajectory entropy gives correlational, not causal, insight.
• Trajectory-aware evaluation adds computational overhead and cost.

Experimental setup

Datasets: tau2-Bench VitaBench HotpotQA 2WikiMultiHopQA

Benchmarks: tau2-Bench VitaBench 2WikiMultiHopQA HotpotQA

Metrics: Avg@4 success rate Avg_Tokens@4 Avg_Steps@4 relative robustness

Baselines: DeepSeek-V3-0324 GPT-4.1 Claude-4.5-Sonnet Gemini-2.5-Pro o3 GPT-5.2

Builds on: tau2-Bench VitaBench HotpotQA

Closest papers by meaning

• AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.91)
• AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.90)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (NeurIPS 2024 · similarity 0.89)
• Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments (ICLR 2026 · similarity 0.89)
• AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.89)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.89)