Research map / Large language models / Agents, code and math
NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NYU CTF Bench, an open benchmark of 200 validated CSAW CTF challenges plus an automated LLM agent framework with function-calling cybersecurity tools, evaluating five LLMs.
Problem
LLMs' ability to autonomously solve Capture-the-Flag cybersecurity challenges has not been thoroughly evaluated, and no scalable open benchmark with automated deployment existed for it.
Approach
A containerized, metadata-rich benchmark of 200 validated CSAW CTF challenges paired with an automated solving framework that gives LLMs function-calling access to domain tools (command execution, file creation, Ghidra decompilation, flag checking, give-up) and automatically scores solutions by flag capture.
Key results
- Claude 3 scored 1500 in the 2022 CSAW Finals, above the median human team score of 1321.
- GPT-4 solved the most categories overall (e.g., 9.80% of rev challenges, 7.69% of pwn), but success rates were low; GPT-4 scored 300 in the 2023 Qualifiers.
- Mixtral and LLaMA 3 solved no challenges, with 100% of their failures classed as wrong answers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT 4 instead of GPT 3.5 | LLM CTF solver · NYU CTF Bench pwn | solved challenges (%): 2.56 -> 7.69 | Helped |
| Use GPT 4 instead of GPT 3.5 | LLM CTF solver · NYU CTF Bench rev | solved challenges (%): 5.88 -> 9.80 | Helped |
| Use GPT 4 instead of GPT 3.5 | LLM CTF solver · NYU CTF Bench web | solved challenges (%): 0 -> 5.26 | Helped |
| Use GPT 3.5 instead of GPT 4 | LLM CTF solver · NYU CTF Bench misc | solved challenges (%): 0 -> 12.5 | Helped |
| Use Claude instead of GPT 4 | LLM CTF solver · NYU CTF Bench crypto | solved challenges (%): 0 -> 5.77 | Helped |
| Use GPT 4 instead of Claude | LLM CTF solver · NYU CTF Bench forensics | solved challenges (%): 0 -> 6.67 | Helped |
| Use GPT 4 instead of GPT 3.5 | LLM CTF solver · CSAW Qual'23 | score: 0 -> 300 | Helped |
Surprising findings
- Open-source models Mixtral and LLaMA 3 solved zero challenges across all categories.
- In 97% of web challenge transcripts the agent failed to retrieve the hint page, the key recon step.
Limitations
- Benchmark sourced from a single CTF series (NYU CSAW), limiting diversity.
- Category imbalance, with some categories like forensics and web underrepresented.
- Only five LLMs evaluated and success rates are low, with results sensitive to randomness at default temperature.
Experimental setup
Datasets: NYU CSAW CTF challenges (2017-2023)
Benchmarks: NYU CTF Bench
Metrics: percentage of challenges solved per category failure type rates CSAW score comparison with human teams
Baselines: GPT-3.5 GPT-4 Claude 3 Mixtral LLaMA 3
Builds on: InterCode-CTF PentestGPT DARPA Cyber Grand Challenge
Closest papers by meaning
- CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence (NeurIPS 2024 · similarity 0.91)
- Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.88)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.87)
- The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning (ICML 2024 · similarity 0.87)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.87)
- On Calibration of LLM-based Guard Models for Reliable Content Moderation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.