Research map / Multi-agent systems and games / LLM agents

PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proteus is a grey-box self-evolving red-team framework that iteratively rewrites agent skill packages using audit, sandbox, and oracle feedback to measure "adaptive leakage" past SkillVetter and AI-Infra-Guard auditors.

Problem

Skill vetting for third-party LLM-agent skills is evaluated with single-shot or static audits, which underestimate residual risk when an attacker can repeatedly revise a skill using auditor findings and runtime feedback.

Approach

A closed-loop LLM mutator edits skills across five axes (objective, topology, code, side-effect channel, documentation) using structured audit findings and sandbox/oracle evidence as in-context feedback, followed by path expansion (alternative implementations) and surface expansion (transfer to new attack objectives) once a successful attack is found.

Key results

• Phase-1 ASR@5 ranges 40–90% across eight (mutator, target, defender) cells; Proteus LCS 0.160 vs ≤0.014 for baselines on SV+GPT-5.4-mini+DeepSeek.
• Phase-2 path/surface expansion yields 438 jointly bypassing and lethal variants out of 999 total; SkillVetter bypassed at ≥93% in every cell.
• Against AI-Infra-Guard, peak joint-success is 41.3%; 50/57 (87.7%) AIG-evolved star skills also bypass SkillVetter without re-mutation.

Surprising findings

• Removing structured findings and leaving only the scalar bypass verdict lowers evasion from 80% to 60% and blocks two seeds from ever succeeding, indicating structured feedback matters beyond scalar reward.
• Restricting attacks to single skills (no chain topology) drops star-rate from 80% to 0% on the ablation cell, so chain topology is load-bearing for evasion.

Limitations

• Seed selection uses the same graph-guided chain composition as the mutator, a potential coupling not checked with holdout seeds.
• Cross-auditor transfer is one-directional (AIG to SkillVetter); symmetric transfer not established.
• Only two auditors, two target backbones, and 20 seeds are evaluated; phase-1 baselines run on a single cell due to token budget.

Experimental setup

Datasets: ClawHub skill corpus (~47k-node skill graph)

Benchmarks: ClawHub-derived 20 malicious seed skills with 10 target repositories

Metrics: ASR@5 learning curve slope strategy diversity growth joint bypass-and-lethal rate audit bypass rate

Baselines: Random Zero-shot Blackbox PyRIT

Builds on: PyRIT RedCodeAgent AutoRedTeamer

Closest papers by meaning

• Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents (NeurIPS 2026 · similarity 0.90)
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · similarity 0.88)
• Stealth Apart, Harm Together: Skill Cascading Attacks on Skill-Based Agent Systems (NeurIPS 2026 · similarity 0.88)
• The Scaling Laws of Skills in LLM Agent Systems (NeurIPS 2026 · similarity 0.87)
• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.87)
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.87)