Research map / Multi-agent systems and games / LLM agents

CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CyberGym, a 1,507-instance benchmark of real-world OSS-Fuzz vulnerabilities where agents reproduce vulnerabilities by generating PoC tests, and reports zero-day and incomplete-patch discoveries.

Problem

Existing AI cybersecurity benchmarks are small-scale and only measure static outcomes, failing to capture the dynamic range of real-world vulnerability analysis.

Approach

A large-scale, execution-verified benchmark built from OSS-Fuzz/ARVO vulnerabilities with four difficulty levels; success requires a PoC that crashes the pre-patch build under sanitizers but not the post-patch build. Unintended PoCs are also mined for zero-days and incomplete patches.

Key results

• Claude-Sonnet-4 with OpenHands achieves the best non-thinking success rate of 17.9% on CyberGym.
• GPT-5 with high reasoning effort reaches 22.0% success, up from 7.7% with minimal reasoning.
• Agentic vulnerability discovery across 431 OSS-Fuzz projects yields 34 confirmed zero-day vulnerabilities (7 from GPT-4.1 and 22 from GPT-5 with 4 overlapping, plus 25 from the discovery run per the intro).

Surprising findings

• Specialized SWE-bench-tuned models (SWE-Gym-32B, R2E-Gym-32B, OpenHands-LM-32B) achieve at most 2.0% success, showing poor generalization from SWE-bench.
• Agents sometimes generate PoCs that crash patched post-fix versions, revealing 18 incomplete patches and zero-days.

Limitations

• Scope limited to memory-safety vulnerabilities in C/C++ projects detectable by sanitizers.
• Success rates are low overall, and results depend on specific agent scaffolds and iteration budgets.
• Possible data contamination, though statistical tests showed no significant pre/post-cutoff difference.

Experimental setup

Datasets: OSS-Fuzz ARVO

Benchmarks: CyberGym SWE-bench

Metrics: success rate zero-day count incomplete patch count

Baselines: Claude-Sonnet-4 Claude-3.7-Sonnet GPT-4.1 GPT-5 o4-mini Gemini-2.5-Flash Qwen3-235B-A22B DeepSeek-V3 SWE-Gym-32B R2E-Gym-32B OpenHands-LM-32B Codex CLI

Builds on: SWE-bench ARVO OSS-Fuzz OpenHands EnIGMA Cybench

Closest papers by meaning

• CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities (ICML 2026 · similarity 0.94)
• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.91)
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.90)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.88)
• A New Framework for Cybersecurity Refusals in AI Agents (ICML 2026 · similarity 0.88)
• Dynamic Risk Assessments for Offensive Cybersecurity Agents (NeurIPS 2025 · similarity 0.88)