Research map / Large language models / Agents, code and math
Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting
Yuqiao Meng, Luoxi Tang, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, Zhaohan Xi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace chain-of-thought prompting with standardized operational modules | GPT-5.1 · Playbook Recommendation | Hit@10: 90.5 -> 92.7 | Improved |
| replace chain-of-thought prompting with standardized operational modules | GPT-5.1 · Security Control Adjustment | Similarity: 81.5 -> 91.0 | Improved |
| replace chain-of-thought prompting with standardized operational modules | GPT-5.1 · Patch Code Generation | Pass rate: 77.6 -> 88.7 | Improved |
| replace chain-of-thought prompting with standardized operational modules | GPT-5.1 · Patch Tool Suggestion | Hit@10: 91.4 -> 98.2 | Improved |
| replace tree-of-thought prompting with standardized operational modules | GPT-5.1 · Advisory Correlation | Hit@10: 85.5 -> 93.6 | Improved |
| replace in-context learning (5 examples) with standardized operational modules | GPT-5.1 · Patch Code Generation | Pass rate: 59.7 -> 88.7 | Improved |
| increase in-context examples from five to ten | GPT-5.1 · Playbook Recommendation | Hit@10: 74.4 -> 75.8 | No clear effect |
| replace chain-of-thought prompting with standardized operational modules | Grok-4.1 · Advisory Correlation | Hit@10: 68.5 -> 81.6 | Improved |
| replace chain-of-thought prompting with standardized operational modules | SevenLLM-7B · Security Control Adjustment | Similarity: 68.4 -> 80.1 | Improved |
| replace chain-of-thought prompting with standardized operational modules | Lily-Cybersecurity-7B · Patch Code Generation | Pass rate: 24.5 -> 29.7 | Improved |
| replace tree-of-thought with chain-of-thought prompting | GPT-5.1 · Security Control Adjustment | Similarity: 86.9 -> 81.5 | Hurt |
| add token-level noise to 10% of input | CyberTeam pipeline · noisy threat logs | performance: less than 5% performance drop | Hurt |
Closest papers by meaning
- Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.87)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.87)
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.86)
- Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.86)
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.86)
- CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
