Research map / Multi-agent systems and games / LLM agents
VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end threat model for visual prompt injection against computer-use and browser-use agents and releases VPI-Bench, a 306-case interactive benchmark across five platforms, showing high attack rates and limited defense effectiveness.
Problem
Computer-use agents with system-level access can be hijacked by malicious instructions rendered visually on web pages, leading to unauthorized actions or privacy leakage, yet prior red-teaming focused on text/HTML attacks and browser-only agents.
Approach
Visual prompt injections embedded in rendered webpages (popups, chat messages, emails) are paired with benign user tasks in sandboxed, interactive replicas of popular platforms, and attacks are judged end-to-end (attempted and completed) with LLM majority voting over agent logs.
Key results
- Computer-use agent Sonnet-3.7 reaches up to 47.8% attempted and 46.2% success on Messenger; Sonnet-3.5 up to 53.9% attempted on Messenger.
- Browser-use agents reach 100% attempted and up to 100% success on Amazon, Booking, and BBC for Sonnet-3.7 and DeepSeek-v3 variants.
- System prompt defense does not reliably reduce attack success; the majority-voting judge reaches 98% attempted and 95% completed accuracy against human labels on 100 logs.
Surprising findings
- System prompt defense changes success and attempted rates inconsistently, sometimes increasing them for some platform-model pairs.
- Sonnet-3.7 is more vulnerable to early injections while Sonnet-3.5 is more vulnerable to late injections.
Limitations
- Assumes unsupervised delegation; users who see malicious popups may intervene.
- Web platforms are reimplemented synthetic replicas hosted for testing rather than live sites.
- UI-TARS 7B was too weak to complete malicious tasks, so it was only partially evaluated.
Experimental setup
Benchmarks: VPI-Bench
Metrics: attempted rate success rate
Baselines: Browser-Use agent Anthropic Computer-Use agent UI-TARS
Builds on: AgentDojo WebArena SeeAct
Closest papers by meaning
- CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.88)
- HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities (ICLR 2026 · similarity 0.87)
- GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.86)
- Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks (NeurIPS 2026 · similarity 0.85)
- EIA: ENVIRONMENTAL INJECTION ATTACK ON GENERALIST WEB AGENTS FOR PRIVACY LEAKAGE (ICLR 2025 · similarity 0.85)
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.