Research map / Multi-agent systems and games / LLM agents
Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares ten professional penetration testers and six existing AI agent scaffolds, plus a new multi-agent scaffold ARTEMIS, on a live ~8,000-host university network.
Problem
Existing AI cybersecurity benchmarks (CTFs, CVE reproduction) lack the realism of live enterprise environments, and no comprehensive comparison of AI agents against human penetration testers in production exists.
Approach
ARTEMIS is a multi-agent penetration-testing scaffold with a supervisor, unlimited parallel sub-agents with dynamically generated expert system prompts, session-based context management, and a triage module that verifies vulnerabilities before submission; it is evaluated head-to-head against human professionals in a live network.
Key results
- ARTEMIS placed second overall with 9 valid vulnerabilities, an 82% valid submission rate, and outperformed 9 of 10 human participants (total score 95.2 vs top human 111.4).
- ARTEMIS A1 (GPT-5) cost $18.21/hour versus $59/hour for A2 and about $60/hour for professional penetration testers.
- Codex (GPT-5), CyAgent (Claude Sonnet 4), and CyAgent (GPT-5) underperformed most human participants; Claude Code and MAPTA refused the task and Incalmo stalled, each with 0 findings.
Surprising findings
- Claude Code and MAPTA refused the offensive task out of the box, yet ARTEMIS using the same underlying models did not refuse, suggesting scaffolding and prompting decisions bypass refusals.
- ARTEMIS produced more false positives than human participants, and it struggled with GUI-based interaction, missing a critical TinyPilot RCE that most humans found.
Limitations
- Compressed time frame (up to 10 hours of active engagement over 4 days) versus typical 1-2 week penetration tests.
- IT team was aware of the test and manually approved flagged actions, lacking authentic defensive conditions.
- Small sample sizes preclude hypothesis testing with statistical power.
Experimental setup
Datasets: Cybench
Benchmarks: Cybench live university network penetration test (~8,000 hosts, 12 subnets)
Metrics: valid findings count validity rate severity score technical complexity score total score cost per hour elicitation pass@1
Baselines: OpenAI Codex Claude Code CyAgent Incalmo MAPTA ten human professional penetration testers
Builds on: Codex CyAgent MAPTA Cybench Alloy Agents
Closest papers by meaning
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.86)
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.86)
- A New Framework for Cybersecurity Refusals in AI Agents (ICML 2026 · similarity 0.86)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.85)
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.85)
- Dynamic Risk Assessments for Offensive Cybersecurity Agents (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.