Research map / Multi-agent systems and games / LLM agents
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Cybench, an open-source benchmark of 40 professional-level CTF tasks with subtasks and first-solve-time difficulty grounding, and evaluates 8 LMs and 4 agent scaffolds on it.
Problem
Existing cybersecurity benchmarks for LM agents are mostly high-school or university-level, closed-source, or have subjectively assigned difficulty, so they do not measure professional-level offensive capability objectively.
Approach
Build a unified framework that turns recent professional CTF challenges into executable tasks with solution scripts, grounds difficulty in human first solve time, and adds subtasks that give partial credit and intermediate-step evaluation.
Key results
- Claude 3.5 Sonnet achieves 17.5% unguided performance on the structured bash agent, and GPT-4o reaches 17.5% subtask-guided performance; OpenAI o1-preview has the highest subtask performance at 46.8%.
- Agents solve none of the tasks with first solve time above 11 minutes unguided, while 73% of tasks with FST up to 11 minutes have non-zero success.
- With pseudoterminal and web search scaffolds, Claude 3.5 Sonnet's unguided performance rises to 20.0%, while GPT-4o's drops to 10.0% and 15.0% respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove Reflection, Plan and Status, Thought, and Log fields (ablation) | Claude 3.5 Sonnet structured bash agent · Cybench unguided | success rate: 17.5% -> 15.0% | Hurt |
| provide pseudoterminal access (ablation) | Claude 3.5 Sonnet structured bash agent · Cybench unguided | success rate: 17.5% -> 20.0% | Helped |
| provide web search (ablation) | Claude 3.5 Sonnet structured bash agent · Cybench unguided | success rate: 17.5% -> 20.0% | Helped |
| provide pseudoterminal access (ablation) | Claude 3.5 Sonnet structured bash agent · Cybench subtask-guided | success rate: 22.5% -> 27.5% | Helped |
| provide pseudoterminal access (ablation) | GPT-4o structured bash agent · Cybench unguided | success rate: 17.5% -> 10.0% | Hurt |
| provide web search (ablation) | GPT-4o structured bash agent · Cybench unguided | success rate: 17.5% -> 15.0% | Hurt |
| remove Reflection, Plan and Status, Thought, and Log fields (ablation) | GPT-4o structured bash agent · Cybench unguided | success rate: 17.5% -> 12.5% | Hurt |
| provide pseudoterminal access (ablation) | GPT-4o structured bash agent · Cybench subtask | fraction of subtasks solved: 40.1% -> 27.1% | Hurt |
| provide web search (ablation) | GPT-4o structured bash agent · Cybench subtask | fraction of subtasks solved: 40.1% -> 42.1% | Helped |
Surprising findings
- Agent scaffolding effects are model-dependent: the pseudoterminal and web search scaffolds help Claude 3.5 Sonnet but hurt GPT-4o.
- Subtask guidance did not enable any model to complete the hardest task (Robust CBC, 24h54m FST), even with length-extension subtask hints.
Limitations
- Only 40 tasks from 4 competitions; difficulty measured by first solve time, which is noisy for subtask-guided runs since competitors lacked subtasks.
- Single attempt for main results and small task count give high variance (e.g., 2.5% per task granularity).
- Possible train-test overlap, notably for Claude 3.5 Sonnet whose knowledge cutoff is April 2024.
Experimental setup
Datasets: HackTheBox cyber-apocalypse-2024 CTF tasks SekaiCTF 2022-23 tasks Glacier CTF tasks HKCert CTF tasks
Benchmarks: Cybench
Metrics: unguided performance subtask-guided performance subtask performance first solve time submission success rate
Baselines: InterCode-CTF NYU CTF Dataset Structured bash agent (unguided)
Builds on: ReAct Reflexion MLAgentBench InterCode-CTF
Closest papers by meaning
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.90)
- EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (ICML 2025 · similarity 0.90)
- CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.89)
- CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios (NeurIPS 2026 · similarity 0.89)
- Cyber-Zero: Training Cybersecurity Agents without Runtime (ICLR 2026 · similarity 0.88)
- KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.