Research map / Multi-agent systems and games / LLM agents
AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AGENTSAFE, a simulator-based benchmark (AI2-THOR sandbox, 9,900 hazardous/normal instructions, three-stage perception-planning-execution diagnostics) for safety of VLM-driven embodied agents, plus a thought-level auditing module SAFE-AUDIT.
Problem
Existing safety benchmarks for embodied VLM agents cover narrow hazard types and only assess final outcomes, obscuring where in the perception-planning-execution pipeline safety failures arise.
Approach
A simulator sandbox with a universal adapter connecting VLM outputs to AI2-THOR actions, a risk-aware instruction suite (human, environment, self harm, plus jailbreak-enhanced variants), and stage-wise metrics (GR, HR, PRR, PSR, TSR) with a thought-level auditor (SAFE-AUDIT) that refines the initial reasoning before planning.
Key results
- GPT-5-mini reaches 81.51% GR on normal instructions, while Claude-sonnet-3.5 achieves 90.11% PRR on Human-Harm tasks and Claude-opus-4 85.56%.
- Step-v1-8k has zero PRR on Self-Harm and Env-Harm tasks with PSR above 72%, indicating near-total failure to refuse those risky instructions.
- SAFE-AUDIT lowers PSR and TSR on hazardous instructions to averages of 3.52% and 0.48%, and raises average TSR on normal instructions by 2.22%.
Surprising findings
- Multilingual jailbreaks improve PSR for Env-Harm but reduce TSR, as jailbreak rewording impairs instruction clarity needed for embodied execution.
- ProgPrompt has zero PRR across all harmful categories, while ReAct rejects over half of Human-Harm instructions.
Limitations
- Evaluation is limited to simulated AI2-THOR environments without sim-to-real validation.
- Adversarial instructions are generated only by six existing jailbreak methods.
Experimental setup
Datasets: AI2-THOR scenes SAFE-VERSE instructions
Benchmarks: SAFE-VERSE
Metrics: grounding recall hallucination rate planning rejection rate planning success rate task success rate
Baselines: AgentSpec ThinkSafe ReAct ProgPrompt
Builds on: AI2-THOR ReAct ProgPrompt AgentSpec
Closest papers by meaning
- IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks (AAAI 2026 · similarity 0.93)
- SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents (NeurIPS 2026 · similarity 0.90)
- HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments (ICLR 2024 · similarity 0.89)
- HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents (ICLR 2025 · similarity 0.88)
- ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs (NeurIPS 2026 · similarity 0.88)
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.