Research map / Multi-agent systems and games / LLM agents

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CyberGym-E2E, a 920-task benchmark built by an agent-assisted pipeline from OSS-Fuzz vulnerabilities that evaluates AI coding agents across vulnerability discovery (PoC generation), patching, and functionality tests.

Problem

Existing cybersecurity benchmarks for AI agents are small, cover only part of the vulnerability lifecycle (detection, PoC, or patching), lack realistic agent environments, or have weak patch-validity tests.

Approach

An automated, agent-enhanced pipeline turns historical OSS-Fuzz vulnerabilities into build environments where agents must find the bug, craft a sanitizer-triggering PoC, and patch it, validated by developer-written unit tests and a multi-stage crash/test oracle.

Key results

• Best patch-only success is 82.3% (Opus 4.5, Claude Code) but end-to-end S3 is only 19.2% on 615 tasks.
• On the expanded 920-task benchmark, GPT-5.4 with Codex reaches 65.9% S3 and 22.2% S4.
• Cross-run feedback improves Opus 4.5 end-to-end S3 from 34.1% to 41.2% in the uncapped-cost analysis.

Surprising findings

• Vulnerability discovery, not patch generation, is the main bottleneck; many valid patches fix a different vulnerability than the ground truth (S3 much higher than S4).
• No significant memorization effect was found for pre- vs post-knowledge-cutoff vulnerabilities.

Limitations

• Focuses only on memory-safety bugs in C/C++ projects with sanitizer-based oracles.
• Human expert validation of test coverage remains the most time-intensive step.
• Shallow patches that pass validation are counted as successes.

Experimental setup

Datasets: OSS-Fuzz ARVO CyberGym

Benchmarks: CyberGym-E2E

Metrics: success rate at stages S1-S4 patch-only success rate

Baselines: CyberGym BountyBench SEC-bench SeCodePLT SecureAgentBench SecRepoBench PatchAgent AutoPatchBench PrimeVul

Builds on: OSS-Fuzz ARVO CyberGym

Closest papers by meaning

• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.94)
• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.91)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.89)
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.87)
• A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle (NeurIPS 2026 · similarity 0.86)
• CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.86)