Research map / Multi-agent systems and games / LLM agents

CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CVE-Factory, a six-stage multi-agent (Claude Code agents with an orchestrator) pipeline that turns sparse CVE metadata into executable Terminal-Bench-style security tasks, releases the LiveCVEBench benchmark (190 tasks), and fine-tunes Qwen3-32B on distilled trajectories.

Problem

Evaluating and training code agents on security vulnerability repair requires executable, up-to-date CVE tasks, but manual reproduction of CVEs is costly (10+ hours each) and unscalable, and prior automated frameworks are limited to Python or well-configured repositories.

Approach

Decouple CVE task construction into three independent generation stages (information collection, file generation, environment construction) followed by three progressive coupling verification stages (environment vs tests, solution vs environment, holistic validation), with an orchestrator that routes pause signals back to original file creators and relies on objective script-based checks.

Key results

• On LiveCVEBench, Qwen3-32B fine-tuned on 4k CVE-Factory trajectories rises from 5.29% to 35.79% pass rate, and Terminal-Bench from 12.50% to 31.25% (3k) / 28.75% (4k).
• CVE-Factory reaches 95.35% solution correctness and 96.13% environment fidelity against PatchEval expert reproductions on 215 CVEs.
• On 454/554 real-world CVEs from May-Dec 2025, CVE-Factory achieves a 66.2% manually verified success rate.

Surprising findings

• Claude Code single-agent baseline completes 0% of CVE reproductions, while the decoupled multi-agent pipeline reaches 90.3% on a 31-CVE sample.
• Nearly all evaluated LLMs perform worse on post-release CVEs than pre-release CVEs, suggesting possible contamination or distribution shift.

Limitations

• Agents sometimes fall back to mock implementations and static tests despite explicit constraints (94 mock cases among false positives in real-world evaluation).
• Complex multi-container orchestration (3+ containers) remains a bottleneck for environment synthesis.
• Many CVEs are irreproducible due to proprietary software or hardware; 66.2% verified success is after excluding objectively irreproducible cases.

Experimental setup

Datasets: CVElistV5 PatchEval SETA

Benchmarks: LiveCVEBench PatchEval Terminal-Bench

Metrics: pass rate solution correctness environment fidelity test quality verified success rate

Baselines: Claude Code CVE-Genie SETA Claude Sonnet 4.5 Claude Opus 4.5 Qwen3-Coder-480B MiniMax-M2

Builds on: Terminal-Bench Claude Agent SDK PatchEval CVE-Genie

Closest papers by meaning

• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.89)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.89)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.89)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.88)
• Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems (AAAI 2026 · similarity 0.87)
• EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (ICML 2025 · similarity 0.87)