AI Paper Atlas

Research map / Large language models / Agents, code and math

ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar

ICML 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
remove alert logs from the database (raw only) (ablation)GPT-4o baseline agent · ExCyTIn-Benchaverage reward: 0.260 -> 0.213Hurt
use an alert-only database (ablation)GPT-4o baseline agent · ExCyTIn-Benchaverage reward: 0.260 -> 0.459Improved
use full-history database instead of per-incident slices (ablation)GPT-4o baseline agent · ExCyTIn-Benchaverage reward: 0.260 -> 0.248Hurt
add ReAct few-shot examplesGPT-4o · ExCyTIn-Bench (k=1)average reward: 0.26 -> 0.354Improved
add Expel (distilled rules and retrieved examples)GPT-4o · ExCyTIn-Bench (k=1)average reward: 0.26 -> 0.390Improved
add ReAct promptingo3-mini · ExCyTIn-Bench (k=1)average reward: 0.219 -> 0.25No clear effect
add Reflection on top of ReAct (3 trials) instead of Best-of-NGPT-4o-mini · ExCyTIn-Bench (k=3)average reward: 0.423 -> 0.452Improved
add Reflection on top of ReAct + BoNGPT-4o · ExCyTIn-Bench (k=3)average reward: 0.563 -> 0.563No clear effect
add Strategy prompt (use alert tables as reference)GPT-4o-mini · ExCyTIn-Bench (k=1)average reward: 0.165 -> 0.290Improved
use Expel instead of Strategy (cost)GPT-4o · ExCyTIn-Bench (k=1)cost: 0.18 -> 0.38Hurt
increase max turns from 5 to 15, then to 25 (ablation)baseline agent · ExCyTIn-Benchreward: spikes from 5 to 15 turns, plateau 15 to 25Mixed
use Best-of-N with 3 trials on ReActGPT-4o · ExCyTIn-Benchaverage reward: 0.354 -> 0.563Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy