Research map / Large language models / Agents, code and math
CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SecKnowledge, a ~400k-instruction cybersecurity dataset from schema-driven expert parsing of MITRE/BRON/SIEM/Sigma data plus hybrid Evol/Self-Instruct content-grounded generation, fine-tunes CyberPal.AI models on Llama-3-8B, Mistral-7B and Phi-3-medium, and introduces SecKnowledge-Eval with 14 benchmarks.
Problem
General LLMs underperform on complex, domain-specific cybersecurity tasks because generic instruction data does not capture security expertise and relationships across security frameworks.
Approach
Generate instruction data from structured security resources (MITRE ATT&CK, CWE, CVE, CAPEC, BRON, SIEM/Sigma rules) using expert-designed schemas and parsers, then expand it with a hybrid Evol-Instruct/Self-Instruct content-grounded pipeline that routes between the two methods over iterations, with an LLM evaluator filtering outputs.
Key results
- CyberPal.AI-Llama improves the training-aligned average from 52.54 to 74.70 over Meta-Llama-3-8B-Instruct, and CyberPal.AI-Mistral from 52.02 to 76.41 over Mistral-7B-Instruct-v0.3.
- On public benchmarks, CyberPal.AI-Phi averages 77.27 vs 66.78 for Phi-3-medium-4k-instruct, and CyberPal.AI-Llama averages 74.47 vs 64.07 for the Llama-3 baseline.
- On the adversarial MITRE ATT&CK set, CyberPal.AI-Mistral drops 6% from original accuracy versus 14% for Mistral-7B-Instruct-v0.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Mistral-7B on SecKnowledge | Mistral-7B-Instruct-v0.3 · Training-aligned tasks (average) | average score: 52.02 -> 76.41 | Helped |
| Fine-tune Llama-3-8B on SecKnowledge | Meta-Llama-3-8B-Instruct · Training-aligned tasks (average) | average score: 52.54 -> 74.70 | Helped |
| Fine-tune Phi-3-medium on SecKnowledge | Phi-3-medium-4k-instruct · Training-aligned tasks (average) | average score: 57.84 -> 75.10 | Helped |
| Fine-tune Mistral-7B on SecKnowledge | Mistral-7B-Instruct-v0.3 · Public cyber-security benchmarks | average score: 60.99 -> 69.92 | Helped |
| Fine-tune Llama-3-8B on SecKnowledge | Meta-Llama-3-8B-Instruct · Public cyber-security benchmarks | average score: 64.07 -> 74.47 | Helped |
| Fine-tune Phi-3-medium on SecKnowledge | Phi-3-medium-4k-instruct · Public cyber-security benchmarks | average score: 66.78 -> 77.27 | Helped |
| Fine-tune with SecKnowledge | CyberPal.AI models · Adversarial MITRE ATT&CK | accuracy degradation from original MITRE ATT&CK: Mistral: 14% -> 6%; Llama: 19% -> 9%; Phi: 13% -> 7% | Helped |
Surprising findings
- Fine-tuned models improved on public benchmarks they were never trained on, with average gains of 9-10% across general cyber-security datasets.
- Pipeline uses no language model for BRON path construction; full synthetic generation was avoided for BRON and Sigma rule generation due to hallucination risk.
Limitations
- Synthetic generation depends on Mixtral and LLM evaluators, which may introduce errors into security instructions.
- Full synthetic generation was not applicable to BRON or Sigma rule generation, so those parts depend on hand-built parsers.
- Many SecKnowledge-Eval tasks are training-aligned and built by the same authors from the same sources.
Experimental setup
Datasets: MITRE ATT&CK CWE CVE CAPEC BRON SIEM QRadar rules to TTP mappings Sigma rules Security Wiki Security Stack Exchange security interview Q&A threat reports CyberBench-style CTI sources
Benchmarks: SecKnowledge-Eval Adversarial MITRE ATT&CK SIEM Rule TTP Mapping CTI Detection and Mitigation Mapping CWE Technical Impact Mapping CWE Description Summarization CTI Relationship Prediction CTI Entity Classification CISSP Assessment SecMMLU CyberMetric CTI-MCQ
Metrics: accuracy ROUGE-1 ROUGE-2 ROUGE-L
Baselines: Mistral-7B-Instruct-v0.3 Meta-Llama-3-8B-Instruct Phi-3-medium-4k-instruct Gemma-2-2b Gemma-2-9b
Builds on: Evol-Instruct Self-Instruct BRON Vicuna
Closest papers by meaning
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.86)
- CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence (NeurIPS 2024 · similarity 0.85)
- KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards (NeurIPS 2026 · similarity 0.84)
- CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs (AAAI 2024 · similarity 0.84)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.84)
- HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.