Research map / Large language models / Agents, code and math

AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents

NeurIPS 2025 · Large language models › Agents, code and math · approach: Software engineering agents · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AgentAuditor, a training-free memory-augmented RAG framework that lets LLM evaluators judge agent safety and security using retrieved CoT experiences, and releases the ASSE-Bench benchmark (2293 records).

Problem

Automated evaluators (rule-based and LLM-based) miss step-wise agent risks, cumulative harms, and ambiguous safety boundaries, and fail to reach human-level agreement when judging agent safety and security.

Approach

Build an experiential memory by having the LLM extract structured features (scenario, risk type, failure mode) and CoT traces for representative FINCH-clustered past cases, then retrieve the most relevant CoTs via a two-stage (content top-n, then feature-weighted re-rank) RAG to few-shot the evaluator, with no weight updates.

Key results

• Gemini-2 with AgentAuditor reaches 96.31 F1 and 96.10 accuracy on R-Judge, up from 82.27 F1 and 81.21 accuracy without it.
• AgentAuditor improves F1 for all listed models on all datasets; e.g., Llama-3.1-8B accuracy on ASSE-Security rises from 49.69 to 81.03.
• Gemini-2 with AgentAuditor (96.1 Acc on R-Judge, 93.2 on ASSE-Security) approaches the single-annotator average (95.7 and 94.3) per the authors' human evaluation.

Measured effects reported in the paper

Surprising findings

• CoT alone or few-shot alone without the other component underperforms the base model; only the combination improves results.
• Gemini-2 with AgentAuditor scores 96.31 F1 on R-Judge, which is 15.1 points above ShieldAgent per the paper.

Limitations

• Performance depends on label quality of the memory shots; label defects reduce gains.
• Relies on heuristic weights and parameters chosen empirically; universality not verified.
• Benchmark is English-only and binary safe/unsafe labels only.

Experimental setup

Datasets: R-Judge AgentSafetyBench AgentSecurityBench AgentDojo AgentHarm

Benchmarks: R-Judge ASSE-Security ASSE-Safety ASSE-Strict ASSE-Lenient AgentHarm

Metrics: F1 accuracy recall human-level comparison

Baselines: R-Judge prompt baseline (no AgentAuditor) ShieldAgent Llama-Guard-3 Auto-CoT-style all-samples few-shot

Builds on: chain-of-thought prompting Auto-CoT FINCH clustering R-Judge prompt template

Closest papers by meaning

• GuardAgent: Safeguard LLM Agents via Knowledge-Enabled Reasoning (ICML 2025 · similarity 0.90)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.89)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.89)
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.89)
• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.88)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data (ICLR 2026 · similarity 0.88)