Research map / Multi-agent systems and games / LLM agents

HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HackWorld, a CTF-based benchmark of 36 vulnerable web applications for evaluating computer-use agents on exploiting web vulnerabilities through GUI interaction with Kali security tools.

Problem

Computer-use agents may operate in vulnerable web environments, but their ability to discover and exploit web application vulnerabilities through visual interfaces is unmeasured, since existing CUA benchmarks use sanitized environments.

Approach

Wraps 36 CTF web challenges (from NYU CTF Bench, Cybench, InterCode-CTF) in Docker containers inside a Kali Linux environment with security tools, and evaluates off-the-shelf CUAs by flag capture under several observation spaces (screenshot, screenshot+a11y tree, set-of-marks).

Key results

• Best CUA (Claude-3.7-Sonnet) reaches about 11.1% success on the 36 challenges in the screenshot setting, with 10.18% averaged across observation spaces.
• Claude-4-Opus averages 4.63% and Claude-3.5-Sonnet 3.71%; UI-TARS-1.5-7B and Qwen-2.5-VL-72B-Instruct score 0% in most conditions.
• A one-way ANOVA finds no statistically significant difference in success rates across observation spaces (p > 0.1).

Surprising findings

• Newer and larger models (Claude-4-Opus, Claude-4-Sonnet) underperform the older Claude-3.7-Sonnet, challenging the scaling hypothesis for web-security tasks.
• Perception improvements (set-of-marks, a11y tree) do not raise success; the bottleneck is reasoning, planning, and tool orchestration.

Limitations

• Small benchmark of 36 challenges, so success-rate differences are noisy and largely not significant.
• Challenges are drawn from older public CTF competitions and may be memorized or contaminated.
• Failure analysis is largely qualitative and based on manual inspection of trajectories.

Experimental setup

Datasets: NYU CTF Bench Cybench InterCode-CTF

Benchmarks: HackWorld

Metrics: success rate (flag capture) tool usage rate average tools per trajectory

Baselines: Claude-3.5-Sonnet Claude-3.7-Sonnet Claude-4-Sonnet Claude-4-Opus UI-TARS-1.5-7B Qwen-2.5-VL-72B-Instruct

Builds on: OSWorld WebArena Cybench NYU CTF Bench InterCode-CTF

Closest papers by meaning

• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.87)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents (ICLR 2026 · similarity 0.87)
• EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (ICML 2025 · similarity 0.85)
• A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle (NeurIPS 2026 · similarity 0.85)
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.84)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.84)