Research map / Multi-agent systems and games / LLM agents

EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities

ICML 2025 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EnIGMA, a SWE-agent-based LM agent with interactive agent tools (gdb debugger and netcat/pwntools-style server connection), LM and simple summarizers, and demonstrations, reaching state-of-the-art CTF solve rates on NYU CTF, InterCode-CTF and CyBench, and analyzes data leakage and "soliloquizing".

Problem

LM agents have had limited success solving cybersecurity Capture The Flag challenges because existing agent interfaces cannot run interactive terminal tools (debuggers, remote server connections) and cannot handle very long tool outputs.

Approach

Non-blocking Interactive Agent Tools that open nested REPL sessions (gdb debugger, server connection via pwntools) running in parallel with the main shell, plus LM-based and simple output summarizers, built on the SWE-agent interface.

Key results

• EnIGMA with Claude 3.5 Sonnet solves 13.5% (27/200) of NYU CTF versus 4.0% for the previous best NYU agent with the same model.
• On InterCode-CTF, EnIGMA with GPT-4 Turbo reaches 72.0% solved versus 43.0% for the prior best Google DeepMind agent.
• Ablating interactive agent tools lowers overall pass@1 from 29.5% to 27.4% on the aggregated benchmarks with Claude 3.5 Sonnet; removing demonstrations drops it to 23.3%.

Measured effects reported in the paper

Surprising findings

• Claude 3.5 Sonnet frequently 'soliloquizes', generating its own fabricated observations and sometimes the flag without interacting with the environment (48.4% of NYU CTF trajectories affected).
• Removing demonstrations improves web and misc category results, and ablating interactive tools improves web performance.

Limitations

• Agents rarely give up early and mostly exhaust the cost budget on failures.
• Evaluated only on proprietary and one open model with temperature 0; results depend on model-specific behavior such as soliloquizing.
• Interactive tools for web navigation are absent, so web category does not benefit.

Experimental setup

Datasets: NYU CTF InterCode-CTF CyBench HackTheBox challenges (collected) CSAW competition challenges 2013-2016 (development set)

Benchmarks: NYU CTF InterCode-CTF CyBench HTB (collected)

Metrics: % solved (pass@1) average cost per solved instance

Baselines: NYU CTF agent CyBench agent InterCode-CTF agent Google DeepMind agent

Builds on: SWE-agent ReAct NYU CTF benchmark InterCode

Closest papers by meaning

• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.90)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.89)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.88)
• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.87)
• Identifying the Risks of LM Agents with an LM-Emulated Sandbox (ICLR 2024 · similarity 0.87)
• Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.87)