Research map / Multi-agent systems and games / LLM agents
CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CVE-Bench, a sandboxed benchmark of 40 critical-severity real-world web application CVEs with reference exploits and automatic grading across eight standard attack types, and evaluates three LLM agent frameworks with gpt-4o.
Problem
Existing benchmarks for LLM agent cyberattack capability either rely on abstracted CTF challenges or small/short code snippets and do not cover real-world critical web application vulnerabilities with diverse attack types.
Approach
A containerized sandbox per CVE with a reference exploit and an automatic evaluation server that grades eight standard attack goals (DoS, file access, file creation, DB modification, DB access, unauthorized admin login, privilege escalation, outbound service), under zero-day and one-day settings.
Key results
- T-Agent exploits up to 10% of CVEs under zero-day and up to 13% (12.5% reported in text) under one-day settings with five attempts.
- Cy-Agent achieves only 2.5% success in the one-day setting with five attempts.
- ZAP identifies 0 of the CVEs and T-Agent with Llama 3.1 exploits 0 CVEs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the hierarchical multi-agent T-Agent framework instead of the Cybench agent | LLM agents evaluated on CVE-Bench · CVE-Bench one-day, 5 attempts | success rate: 2.5% to 13% | Helped |
| Provide a high-level vulnerability description (one-day instead of zero-day) | LLM agents on CVE-Bench · CVE-Bench | success rate: 10% ... zero-day ... 12.5% ... one-day | Helped |
| Use AutoGPT instead of other evaluated agent frameworks | LLM agents on CVE-Bench · CVE-Bench | success rate | Mixed |
| Enable the SQL team to use sqlmap | T-Agent · CVE-Bench SQL injection exploits | success rate | Helped |
| Use T-Agent with Llama 3.1 rather than GPT-4o | T-Agent on CVE-Bench · CVE-Bench, 5 runs | CVEs exploited: 0 CVEs | Hurt |
Surprising findings
- AutoGPT achieved a higher zero-day success@5 than its one-day success@5, as it found easier unrelated vulnerabilities.
- The open-source Llama 3.1 powered T-Agent exploited zero CVEs, versus GPT-4o.
Limitations
- Only the eight predefined standard attacks are graded, so other successful attacks may be false negatives.
- Covers only 40 web-related CVEs published in a narrow date range (May to June 2024).
- Exploit reproductions are kept private to mitigate contamination, limiting public verifiability.
Experimental setup
Datasets: National Vulnerability Database CVEs
Benchmarks: CVE-Bench
Metrics: success rate (success@1) success@5 token cost monetary cost
Baselines: Cy-Agent AutoGPT ZAP Llama 3.1
Builds on: Cybench ReAct AutoGPT inspect_ai
Closest papers by meaning
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.91)
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.91)
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.91)
- CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities (ICML 2026 · similarity 0.91)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.89)
- CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.