Research map / Multi-agent systems and games / LLM agents

Agent Security is a Systems Problem

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing agent security should be treated as a systems problem with the LLM as an untrusted component, proposing instruction-data separation, verifiable least-privilege policy generation, and information flow control, illustrated by analyzing real-world agent attacks.

Problem

Model-centric robustness approaches to prompt injection and agent attacks are argued to be insufficient; agent security lacks systems-level principles such as least privilege, complete mediation, and information flow control.

Approach

Treat the model powering an agent as an untrusted component and enforce security invariants at the system level using classic systems security principles (instruction-data separation, verifiable least-privilege policies, IFC), with heterogeneous enforcement for defense-in-depth.

Key results

• Analyzes eleven real-world attacks on agentic systems (e.g., ChatGPT SpAIware, Claude Code exfiltration, Devin secret leaks) and maps each to violated security principles.
• Identifies three open research problems: instruction-data separation, verifiable policy generation, and information flow control for LLMs.

Surprising findings

• Argues that model-based defenses for instruction-data separation will always be evaded by persistent attackers.
• Argues that LLM-based safety agents act as unpredictable probabilistic reference monitors and TCBs.

Limitations

• No empirical evaluation of proposed mechanisms; claims rest on case-study analysis and argument.
• Proposed mechanisms face unresolved challenges (label explosion, declassification, semantic gap).

Experimental setup

Builds on: W xor X memory protection information flow control Content Security Policy Instruction Hierarchy

Closest papers by meaning

• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.87)
• Architecture Matters for Multi-Agent Security (ICML 2026 · similarity 0.87)
• MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems (ICML 2026 · similarity 0.86)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.86)
• Attack Selection In Agentic AI Control Evaluations Meaningfully Decreases Safety (NeurIPS 2026 · similarity 0.86)
• A New Framework for Cybersecurity Refusals in AI Agents (ICML 2026 · similarity 0.86)