Research map / Large language models / Agents, code and math
Toward Cybersecurity-Expert Small Language Models
Matan Levi, Daniel Ohayon, Ariel Blobstein, Ravid Sa, Ian Molloy, Yair Allouche
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SecKnowledge 2.0 instead of original SecKnowledge (ablation) | CyberPal 2.0-4B · Qwen3-4B-base, avg of 9 benchmarks | average accuracy: 68.60 -> 72.68 | Improved |
| add expert-written task formats instead of general-purpose formats (ablation) | SecKnowledge 2.0 pipeline · Qwen3-4B-base, avg of 9 benchmarks | average accuracy: 70.04 -> 72.68 | Improved |
| add retrieval search grounding (ablation) | SecKnowledge 2.0 pipeline · Qwen3-4B-base, avg of 9 benchmarks | average accuracy: 71.11 -> 72.68 | Improved |
| swap reformatting backbone gpt-oss-120B for Llama-4-Maverick (ablation) | SecKnowledge 2.0 pipeline · Qwen3-4B-base, avg of 9 benchmarks | average accuracy: 72.68 -> 72.32 | No clear effect |
| generic reformatting of original SecKnowledge (ablation) | SecKnowledge · Qwen3-4B-base, avg of 9 benchmarks | average accuracy: 68.60 -> 70.04 | Improved |
| fine-tune from base model instead of post-trained model (ablation) | CyberPal 2.0-8B · CTIBench-MCQ/RCM/SecEval avg | average accuracy: 66.47 -> 76.01 | Improved |
| fine-tune from base instead of post-trained model (ablation) | CyberPal 2.0-8B · CTIBench-RCM | accuracy: 65.10 -> 85.95 | Improved |
| fine-tune on SecKnowledge 2.0 (vs post-trained Qwen3-4B) | Qwen3-4B · Average of 9 benchmarks | average accuracy: 65.19 -> 72.68 | Improved |
| fine-tune on SecKnowledge 2.0 (vs gpt-oss-20B) | gpt-oss-20B · CyberMetric 2000 | accuracy: 90.20 -> 89.05 | Hurt |
| fine-tune on SecKnowledge 2.0 (vs gpt-oss-20B) | gpt-oss-20B · Weakness Impact Mapping | accuracy: 71.91 -> 70.77 | Hurt |
| fine-tune on SecKnowledge 2.0 (vs Qwen3-14B) | Qwen3-14B · CyberMetric 2000 | accuracy: 89.85 -> 89.95 | No clear effect |
| fine-tune on SecKnowledge 2.0 (vs Qwen3-4B) | Qwen3-4B · Threat-to-TTP mapping | accuracy: 74.00 -> 82.75 | Improved |
Closest papers by meaning
- CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions (AAAI 2025 · similarity 0.92)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.89)
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.88)
- KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
- Cyber-Zero: Training Cybersecurity Agents without Runtime (ICLR 2026 · similarity 0.86)
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
