Research map / Multi-agent systems and games / LLM agents

Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Defense-as-Skill, implementing a runtime guard (SkillSonar) as an installable skill for skill-augmented agents, evolved via MCTS over guard-skill edits and evaluated on the new SCOPE-R attack dataset.

Problem

Skill-augmented agents load reusable skills that can persistently steer tool-use actions, so malicious skills can trigger unsafe actions only after task and workspace context makes them look useful; pre-install vetting and external guard models are insufficient or hard to deploy.

Approach

Implement the runtime guard itself as an installable, inspectable, editable skill that triages each sensitive action against the user's task boundary (allow/replan/confirm), and optimize that skill on disk with Monte-Carlo Tree Search driven by rollout feedback.

Key results

• On repeated GLM-5 runs (N=10), SkillSonar reduces ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115 on Claude Code.
• On GPT-5.4, SkillSonar achieves ID/OOD ASR of 0.019/0.034 versus 0.588/0.559 for no-guard, but with lower task utility (0.533 ID vs 0.731).
• Explicit guard invocation reduces ID/OOD ASR from 0.400/0.582 to 0.104/0.109 versus ordinary skill selection.

Surprising findings

• Leaving the guard skill to ordinary skill selection provides only limited protection; explicit invocation is needed.
• AcceptEdits's allowlist variant returns ASR close to no-guard levels, showing that relaxing permissions reopens attack paths.

Limitations

• Explicit invocation is a soft protocol; host agent may fail to read or comply with the guard.
• Remaining ASR under adaptive attacks (22.9%) shows the policy is not robust to all adaptive attacks.
• OOD ASR is higher than ID for GLM-5 and Claude Haiku 4.5; GPT-5.4 shows reduced utility.

Experimental setup

Datasets: PinchBench SCOPE-R SkillSafetyBench WildClawBench

Benchmarks: SCOPE-R SkillSafetyBench WildClawBench

Metrics: attack success rate task utility confirmation count token cost

Baselines: No-guard System prompt AcceptEdits AcceptEdits + allowlist AgentSpec TS-Guard

Builds on: Claude Code OpenClaw PinchBench AgentSpec

Closest papers by meaning

• PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.90)
• Stealth Apart, Harm Together: Skill Cascading Attacks on Skill-Based Agent Systems (NeurIPS 2026 · similarity 0.88)
• Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks (NeurIPS 2026 · similarity 0.87)
• Group of Skills: Group-Structured Skill Retrieval for Agent Skill Libraries (NeurIPS 2026 · similarity 0.87)
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.87)
• PragLocker: Protecting Agent Intellectual Property in Untrusted Deployments via Non-Portable Prompts (ICML 2026 · similarity 0.87)