Research map / Large language models / Agents, code and math

KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

NeurIPS 2026 · Large language models › Agents, code and math · approach: Code generation benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces KaliBench, a verified benchmark of 8,504 NL-to-CLI query-command pairs for Kali Linux tools, evaluates 20+ LLMs in three hint settings, and shows SFT and runtime-free GRPO on an 8B model (RedSage-K) approach much larger models.

Problem

LLMs must translate analyst intent into exact, executable cybersecurity CLI commands, but existing evaluations cover knowledge QA or end-to-end CTF agents and do not measure schema-free command generation for real tools.

Approach

A documentation-grounded benchmark of Kali tool query-command pairs with alias-aware component scoring (tool, optional flags, positional args) under three information settings, plus rewards computed from the deterministic command structure without executing model outputs, enabling runtime-free RLVR.

Key results

• Hinted mode raises mean Exact Correct from 22.3% (unrestricted) to 73.1% and Total Score from 59.4% to 90.8%, with argument construction the main bottleneck.
• RedSage-K (SFT+GRPO), an 8B model, reaches 79.2 average Total Score, third among open-weight models and 1.0 point below DeepSeek-V3.2 (80.2).
• GPT-5.6-Sol achieves 61.68% full-set Exact Correct in unrestricted mode, the highest among proprietary systems evaluated.

Surprising findings

• Cybersecurity-tuned models do not consistently beat strong general-purpose models, though they help more in under-specified settings.
• GRPO and SFT+GRPO slightly reduce hinted-mode Total Score relative to the original RedSage-Ins model.

Limitations

• Focus on single-turn single-command generation; no multi-step workflows.
• Data is synthetic and grounded in documentation; over 50% of raw generated commands needed filtering, and residual label errors remain.
• Proprietary model coverage is uneven due to provider safety refusals, complicating comparison.

Experimental setup

Datasets: Kali Linux tool documentation (2,372 tools)

Benchmarks: KaliBench

Metrics: Exact Correct Tool Accuracy Optional-Argument F1 Positional-Argument F1 Total Score

Baselines: Foundation-Sec-Instruct Llama-Primus-Merged RedSage-Ins Qwen3-8B DeepSeek-V3.2 GLM-5.2 GPT-5.6-Sol Claude Opus 5 Codex (GPT-5.5)

Builds on: GRPO LoRA Unsloth SemHash Cyber Kill Chain

Closest papers by meaning

• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.88)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.88)
• Cyber-Zero: Training Cybersecurity Agents without Runtime (ICLR 2026 · similarity 0.86)
• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.86)
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.86)
• Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.86)