Research map / Multi-agent systems and games / LLM agents

SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SENTINEL, a multi-level formal framework that evaluates FM-based embodied agents' physical safety via temporal logic specs at semantic, plan, and trajectory (computation-tree CTL) levels in VirtualHome and AI2-THOR.

Problem

Existing safety evaluations of foundation-model-based embodied agents rely on heuristic rules or subjective FM judges and do not localize where safety violations arise (interpretation, planning, or execution).

Approach

Formalize natural-language safety constraints into LTL/CTL specifications and check them at three progressive levels: LTL-equivalence of generated formulas (semantic), LTL checking of high-level plans (plan), and CTL checking over a computation tree merged from sampled trajectories (trajectory), with counterexamples fed back for refinement.

Key results

• On VirtualHome plan-level, DeepSeek-V3.1 reaches 96.5% safety and 88.8% success&safety, while GPT-5.5 has 90.2% success.
• At trajectory level in AI2-THOR, GPT-5.5 achieves 51.7% success&safety, the highest among listed LLMs, and all agents remain low in absolute terms.
• In a feedback refinement loop with Claude-Opus-4.7, SENTINEL counterexample feedback improves Safe by +10.8% vs +5.2% for LLM-as-Judge feedback, while LLM-as-Judge misses nearly 83% of truly unsafe trajectories.

Surprising findings

• LLM-as-Judge labels 95% of safe trajectories correctly but misses ~83% of unsafe ones, so its feedback is nearly uncorrelated with real hazards.
• Among VLMs, Gemma-3-27B-it attains the highest trajectory-level Safe rate (77.3%) yet only 15.6% success&safety.

Limitations

• Trace-based CTL over sampled trajectories gives empirical detection, not formal guarantees; missed violations remain possible.
• Only symbolic-level atomic propositions supported by current simulators; timed (MTL/TCTL) and fine-grained physical constraints are excluded.
• Real-world evaluation is a single tabletop Franka case study and does not show empirical sim-to-real transfer.

Experimental setup

Datasets: VirtualHome tasks AI2-THOR tasks ALFRED tasks

Benchmarks: VirtualHome safety tasks (91 long-horizon) AI2-THOR safety suite (120 short + 91 long tasks)

Metrics: syntax error rate equivalence rate success rate safety rate success & safety rate

Baselines: LLM-as-Judge feedback GPT-5 Claude-Sonnet-4 Gemini-2.5-Flash DeepSeek-V3.1

Builds on: LTL model checking CTL model checking SafeAgentBench ReAct-style refinement loops

Closest papers by meaning

• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions (CVPR 2026 · similarity 0.90)
• IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks (AAAI 2026 · similarity 0.90)
• PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety (NeurIPS 2026 · similarity 0.88)
• AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.86)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.86)
• Agent MechSuits : Mechanistic Subspace Safety Steering for Multi-Turn CLI Agents (NeurIPS 2026 · similarity 0.86)