Research map / Multi-agent systems and games / LLM agents
SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SEC-bench, a multi-agent framework that automatically builds verified real-world CVE security benchmarks (PoC generation and vulnerability patching) and evaluates LLM code agents on them.
Problem
Existing security benchmarks for LLM agents rely on synthetic challenges or unreliable vulnerability datasets lacking verifiable PoCs and patches, so they fail to reflect real-world security engineering.
Approach
A multi-agent verifier (builder, exploiter, fixer agents coordinated by a manager) automatically reproduces in-the-wild CVEs in Docker environments, using sanitizer reports as the deterministic oracle for PoC validity and patch correctness, and turns verified instances into PoC-generation and patching tasks.
Key results
- Verifier successfully reproduces 200 CVE instances out of 898 candidates at an average cost of $0.87 per instance, an 85.7% higher overall success rate than CodeAct on a 50-instance comparison (26.0% vs 14.0%).
- Best PoC generation success is 18.0% and best vulnerability patching success is 34.0% on the full 200-instance dataset.
- Claude 3.7 Sonnet with SWE-agent achieves 33.8% patch and 12.5% PoC resolve rates on the 80-instance subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the single-agent codeact scaffold with the multi-agent repro scaffold (ablation) | vulnerability verification · 50 randomly selected SEC-bench instances across 23 projects | overall success rate (%): 14.0 -> 26.0 | Helped |
| replace the single-agent codeact scaffold with the multi-agent repro scaffold (ablation) | vulnerability verification · 50 randomly selected SEC-bench instances across 23 projects | builder success rate (%): 72.0 -> 90.0 | Helped |
| replace the single-agent codeact scaffold with the multi-agent repro scaffold (ablation) | vulnerability verification · 50 randomly selected SEC-bench instances across 23 projects | exploiter success rate (%): 33.3 -> 35.6 | Helped |
| replace the single-agent codeact scaffold with the multi-agent repro scaffold (ablation) | vulnerability verification · 50 randomly selected SEC-bench instances across 23 projects | fixer success rate (%): 58.3 -> 81.2 | Helped |
| use post-cutoff rather than pre-cutoff instances | GPT-4o with SWE-agent · PoC generation | resolved rate (%): 6.7 -> 0 | Hurt |
| use post-cutoff rather than pre-cutoff instances | Claude 3 Haiku with SWE-agent · PoC generation | resolved rate (%): 0 -> 0 | No clear effect |
| use post-cutoff rather than pre-cutoff instances | GPT-4o with SWE-agent · vulnerability patching | resolved rate (%): 33.3 -> 40.0 | Helped |
| use post-cutoff rather than pre-cutoff instances | Claude 3 Haiku with SWE-agent · vulnerability patching | resolved rate (%): 20.0 -> 13.3 | Hurt |
Surprising findings
- No significant performance difference before vs after model knowledge cutoff (Wilcoxon p=0.27), suggesting limited contamination effects.
- Agents perform far worse on security tasks than on SWE-bench, with PoC generation at most 18.0% despite >60% on SWE-bench Verified.
Limitations
- Restricted to memory-safety vulnerabilities in C/C++ verifiable by sanitizers
- Manual inspection needed for benchmark quality control
- Evaluation on a subset of 80 instances due to budget constraints
Experimental setup
Datasets: OSV database NVD OSS-Fuzz configurations
Benchmarks: SEC-bench
Metrics: resolved rate (PoC generation) resolved rate (vulnerability patching) builder/exploiter/fixer success rate cost
Baselines: CodeAct SWE-agent OpenHands Aider
Builds on: SWE-agent OpenHands Aider
Closest papers by meaning
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.91)
- CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.91)
- MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.90)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.90)
- SecureClaw: Clawing Back Control of LLM Agents (NeurIPS 2026 · similarity 0.89)
- CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.