Research map / Large language models / Agents, code and math
ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove alert logs from the database (raw only) (ablation) | GPT-4o baseline agent · ExCyTIn-Bench | average reward: 0.260 -> 0.213 | Hurt |
| use an alert-only database (ablation) | GPT-4o baseline agent · ExCyTIn-Bench | average reward: 0.260 -> 0.459 | Improved |
| use full-history database instead of per-incident slices (ablation) | GPT-4o baseline agent · ExCyTIn-Bench | average reward: 0.260 -> 0.248 | Hurt |
| add ReAct few-shot examples | GPT-4o · ExCyTIn-Bench (k=1) | average reward: 0.26 -> 0.354 | Improved |
| add Expel (distilled rules and retrieved examples) | GPT-4o · ExCyTIn-Bench (k=1) | average reward: 0.26 -> 0.390 | Improved |
| add ReAct prompting | o3-mini · ExCyTIn-Bench (k=1) | average reward: 0.219 -> 0.25 | No clear effect |
| add Reflection on top of ReAct (3 trials) instead of Best-of-N | GPT-4o-mini · ExCyTIn-Bench (k=3) | average reward: 0.423 -> 0.452 | Improved |
| add Reflection on top of ReAct + BoN | GPT-4o · ExCyTIn-Bench (k=3) | average reward: 0.563 -> 0.563 | No clear effect |
| add Strategy prompt (use alert tables as reference) | GPT-4o-mini · ExCyTIn-Bench (k=1) | average reward: 0.165 -> 0.290 | Improved |
| use Expel instead of Strategy (cost) | GPT-4o · ExCyTIn-Bench (k=1) | cost: 0.18 -> 0.38 | Hurt |
| increase max turns from 5 to 15, then to 25 (ablation) | baseline agent · ExCyTIn-Bench | reward: spikes from 5 to 15 turns, plateau 15 to 25 | Mixed |
| use Best-of-N with 3 trials on ReAct | GPT-4o · ExCyTIn-Bench | average reward: 0.354 -> 0.563 | Improved |
Closest papers by meaning
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.88)
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.88)
- Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.88)
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.88)
- Monitoring Decomposition Attacks with Lightweight Sequential Monitors (ICLR 2026 · similarity 0.88)
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
