Research map / Multi-agent systems and games / LLM agents

Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Agent-ToM, a Theory-of-Mind-based learning-to-monitor framework that uses a Reason-Verify-Refine inference pipeline and a critique-distilled semantic guardrail memory to detect covert malicious behavior in LLM agents.

Problem

Monitoring autonomous LLM agents for covert, long-horizon malicious behavior (sabotage with hidden side tasks) while existing monitors treat each trajectory independently and do not learn from prior monitoring episodes.

Approach

Structured ToM inference (beliefs, step-level intent hypotheses, expected actions, deviations) followed by a verify-and-refine step, plus a persistent additive semantic guardrail memory distilled from critique signals across training episodes, with no retraining of the underlying model.

Key results

• On SHADE-Arena test, Agent-ToM (external memory Stage 1-2) reaches 0.782 F1 and 0.752 accuracy, above Sequential Ensemble (0.727 F1, 17 calls) and Async Ensemble (0.761 F1, 8 calls).
• On CUA-SHADE-Arena (out-of-distribution), Agent-ToM Stage 1-2 achieves 0.831 F1 and 0.891 accuracy at 1.000 precision.
• Critique feedback improves F1 from 0.662 to 0.814 at full training; self-structured memory beats predefined memory (0.861 vs 0.813 overall F1 on common subset).

Surprising findings

• Async Ensemble reaches near-perfect recall (0.990) but its precision collapses to 0.617, and on CUA its F1 falls to 0.606.
• Agent-ToM Stage 2 (Verify-Refine) gives a larger F1 gain on CUA-SHADE-Arena (+0.216) than on SHADE-Arena (+0.065).

Limitations

• Effectiveness depends on critique quality; noisy diagnostic signals yield misleading guardrails.
• Memory organization schema may need to vary across deployment settings.
• Assumes latent beliefs and intent are inferable from traces, which fails for highly deceptive agents.

Experimental setup

Datasets: SHADE-Arena CUA-SHADE-Arena

Benchmarks: SHADE-Arena CUA-SHADE-Arena

Metrics: F1 Accuracy Recall Precision AUC

Baselines: LLM-Judge (zeroshot) ToM (zeroshot) Agent-Auditor Sequential Ensemble Async Ensemble GEPA prompt-level learning

Builds on: SHADE-Arena Monitor Red Teaming GEPA Agent-Auditor evolving context

Closest papers by meaning

• Constitutional Black-Box Monitoring for Scheming in LLM Agents (ICML 2026 · similarity 0.89)
• AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.88)
• Monitoring Decomposition Attacks with Lightweight Sequential Monitors (ICLR 2026 · similarity 0.88)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.88)
• Reliable Weak-to-Strong Monitoring of LLM Agents (ICLR 2026 · similarity 0.88)
• A-MemGuard: A Proactive Defense Framework For LLM-Based Agent Memory (ICML 2026 · similarity 0.88)