Research map / Large language models / Agents, code and math
CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CTIBench, a suite of five cyber threat intelligence tasks (CTI-MCQ, CTI-RCM, CTI-VSP, CTI-ATE, CTI-TAA) to evaluate zero-shot LLM performance on CTI knowledge and reasoning, testing five models.
Problem
Existing LLM benchmarks are general or narrow cybersecurity tasks focused on memorization; there is no benchmark assessing practical, applied CTI capabilities such as CVE-to-CWE mapping, severity prediction, ATT&CK technique extraction, and threat actor attribution.
Approach
A multi-task benchmark combining a knowledge MCQ dataset generated with GPT-4o from authoritative CTI sources (manually validated) and four practical CTI tasks built from NVD, MITRE ATT&CK, and threat reports, with a graph-search (BFS) alias-based scoring scheme for threat actor attribution that separates correct and plausible answers.
Key results
- ChatGPT-4 (gpt-4-turbo) scored highest on CTI-MCQ (71.0%) and CTI-RCM (72.0%) and CTI-ATE (0.6388 F1), while Gemini-1.5 had the lowest CTI-VSP MAD (1.09).
- LLAMA3-70B matched or beat Gemini-1.5 on three tasks despite being open-weights.
- Models generally performed better on pre-knowledge-cutoff CTI-TAA samples, with ChatGPT-4 correct accuracy dropping from 58.06% to 42.10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an explicit reasoning prompt (ablation) | ChatGPT-4 · CTI-MCQ | accuracy: 71.00 -> 71.84 | No clear effect |
| Add an explicit reasoning prompt (ablation) | ChatGPT-3.5 · CTI-MCQ | accuracy: 54.08 -> 59.16 | Helped |
| Add an explicit reasoning prompt (ablation) | Gemini-1.5 · CTI-MCQ | accuracy: 65.44 -> 65.68 | No clear effect |
| Add an explicit reasoning prompt (ablation) | LLAMA3-70B · CTI-MCQ | accuracy: 65.72 -> 65.48 | No clear effect |
| Add an explicit reasoning prompt (ablation) | LLAMA3-8B · CTI-MCQ | accuracy: 61.32 -> 55.80 | Hurt |
Surprising findings
- Explicit reasoning prompts substantially increased token cost but generally did not improve CTI-MCQ accuracy.
- All models overestimated CVSS scores more often than underestimated them, and performance on CTI-RCM and CTI-VSP declined for the longest descriptions.
Limitations
- Covers only a limited subset of CTI tasks
- Restricted to English-language CTI content
- Risk of malicious misuse of LLMs for generating false threat intelligence
Experimental setup
Datasets: CTI-MCQ NVD CVE records (2024 and 2021) MITRE ATT&CK malware descriptions CTI threat reports from vendors Malpedia aliases
Benchmarks: CTI-MCQ CTI-RCM CTI-VSP CTI-ATE CTI-TAA
Metrics: accuracy mean absolute deviation micro-F1 correct accuracy plausible accuracy
Builds on: MITRE ATT&CK CVSS v3 CWE
Closest papers by meaning
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.91)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.86)
- Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.86)
- KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards (NeurIPS 2026 · similarity 0.85)
- CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions (AAAI 2025 · similarity 0.85)
- On Calibration of LLM-based Guard Models for Reliable Content Moderation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.