Research map / Multi-agent systems and games / LLM agents
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ToolEmu, an LM-emulated sandbox plus LM-based safety and helpfulness evaluators, to find risks of LM agents across many tools; validates with human annotation and benchmarks GPT-4, Claude-2, ChatGPT-3.5 and Vicuna agents.
Problem
Identifying risks of LM agents that use tools is labor-intensive because each tool must be implemented and each test scenario's sandbox set up manually, which limits coverage of long-tail, high-stakes failures.
Approach
Use an LM (GPT-4) to emulate tool execution and sandbox state from tool specifications alone, add an adversarial emulator that instantiates risky sandbox states for red-teaming, and use an LM-based evaluator to score safety and helpfulness of agent trajectories.
Key results
- GPT-4 agent achieves the best safety score of 2.007 with failures in 39.4% of tests; a safety-prompted GPT-4 reaches 2.359 with 23.9% failure incidence.
- Human validation finds about 68-72% precision for identified true failures (exact figures masked in text), with the adversarial emulator detecting about 10 percentage points more true failures than the standard emulator.
- The LM-based safety evaluator agrees with human annotators at a Cohen's kappa comparable to the inter-annotator agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add safety requirements to the prompt (ablation) | GPT-4 agent · Toolemu benchmark | safety score: 2.007 -> 2.359 | Helped |
| Add safety requirements to the prompt (ablation) | GPT-4 agent · Toolemu benchmark | failure incidence: 39.4% -> 23.9% | Helped |
| Add safety requirements to the prompt (ablation) | GPT-4 agent · Toolemu benchmark | helpfulness score: 1.458 -> 1.824 | Helped |
| Add helpfulness requirements to the Safety prompt (ablation) | GPT-4 agent · Toolemu benchmark | safety score: 2.359 -> 2.241 | Hurt |
| Add helpfulness requirements to the Safety prompt (ablation) | GPT-4 agent · Toolemu benchmark | helpfulness score: 1.824 -> 1.624 | Hurt |
| Use the adversarial simulator instead of the standard simulator | Toolemu end-to-end validation · 100 sampled tests | true-failure incidence | Helped |
| Use the adversarial simulator instead of the standard simulator | Toolemu end-to-end validation · 100 sampled tests | failure precision | Hurt |
Surprising findings
- Even the safest evaluated agent (safety-prompted GPT-4) still fails in 23.9% of tests.
- Vicuna agents look safer than ChatGPT-3.5 mainly because they are less effective at using tools, as shown by lower helpfulness scores.
Limitations
- Emulator and evaluator are prompt-engineered GPT-4 and sometimes violate core requirements (critical issues in about 8-14% of trajectories) and the safety evaluator has lower recall than humans.
- Test case curation still relies largely on humans.
- Only the instruction underspecification threat model is studied.
Experimental setup
Datasets: ToolEmu benchmark test cases (curated with GPT-4 and human filtering)
Benchmarks: ToolEmu benchmark
Metrics: safety score helpfulness score failure incidence failure precision Cohen's kappa precision and recall vs human annotators
Baselines: GPT-4 Claude-2 ChatGPT-3.5 Vicuna-1.5-13B Vicuna-1.5-7B NoAct agent
Builds on: ReAct chain-of-thought
Closest papers by meaning
- AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.88)
- AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.88)
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.87)
- RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents (NeurIPS 2025 · similarity 0.87)
- EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (ICML 2025 · similarity 0.87)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.