Research map / Multi-agent systems and games / LLM agents

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Security Arena, a 21-scenario benchmark for adaptive multi-round prompt-injection and disclosure attacks against fresh-session LLM defenders, with a 3x3 frontier-model matrix, controls on provenance, history, and state, frozen replay, and an open competition.

Problem

Static attack pools under-measure LLM-agent security because they do not capture attackers that learn from defender responses across rounds; ASR depends on attacker model, scenario, round budget, and session state.

Approach

An autonomous LLM attacker generates each input live, observes prior defender responses and scorer feedback, and pivots across up to 15 rounds, while each defender response is evaluated in a fresh session; success is judged by executable task-level checks, and the protocol (history, session state, budget) is configurable.

Key results

• Round-1-only scoring yields 0.0%/0.0%/1.0% ASR for Opus/GPT-5.4/Gemini, versus 7.9%/9.5%/16.8% when scoring through 15 rounds.
• A provenance paragraph reduces ASR from 110/270 to 70/270 across six scenarios (Fisher p=3.57e-4), with no change on memleak.
• Pooling three attackers yields 1.7-2.2x as many unique successful inputs as the best single attacker at three times the battle budget.

Surprising findings

• Aggregate ASR hides opposing scenario weaknesses: Opus is compromised far more on memleak (60%) while Gemini is far more compromised on paperaccept (53%).
• Frozen replay transfers poorly across defenders (11.8% cross-target) though same-target reproduction is 67.2%.

Limitations

• Small sample sizes (five battles per frontier cell) and targeted follow-ups chosen after initial results.
• Attacker LLM refusals limit discovered attack space; complements human red-teaming.
• Frontier scaffold has no tools or external state changes; scenarios concentrate on injection and disclosure.

Experimental setup

Datasets: Security Arena scenarios (21 held-out, 10 development)

Benchmarks: Security Arena 21-scenario benchmark

Metrics: attack success rate ASR@15 Wilson 95% interval Fisher exact test Kendall's W

Baselines: PAIR TAP Crescendo AgentDojo AgentHarm

Builds on: PAIR TAP Crescendo AgentBeats AgentDojo

Closest papers by meaning

• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.90)
• AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks (ICML 2026 · similarity 0.89)
• PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.87)
• Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAS (AAAI 2026 · similarity 0.87)
• SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.87)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.87)