Research map / Multi-agent systems and games / LLM agents
EIA: ENVIRONMENTAL INJECTION ATTACK ON GENERALIST WEB AGENTS FOR PRIVACY LEAKAGE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies privacy risks of generalist web agents on compromised websites, proposing Environmental Injection Attack (EIA) that hides persuasive PII-stealing instructions in webpage HTML, evaluated on SeeAct with 177 Mind2Web PII action steps.
Problem
Generalist web agents that complete tasks requiring PII on real websites may leak that PII (or the full user request) if they interact with compromised webpages; this privacy risk is largely unexplored.
Approach
Inject invisible (zero-opacity) form or mirrored HTML elements carrying persuasive instructions into benign webpages, adapted to the page's layout and position, so the web agent types PII into the injected field; auto-submission sends the data out and removes the element. Relaxing opacity to a low non-zero value (Relaxed-EIA) lets the injection also influence the screenshot-based action generation
Key results
- MI injection at position P+1 with GPT-4V reaches up to 70% ASR for stealing specific PII at an action step.
- Relaxed-EIA (opacity 0.2) raises full-request leakage ASR on GPT-4V from 0% (standard EIA) to 16% at position P-3.
- None of the compromised webpages were flagged by VirusTotal, and a defensive system prompt did not meaningfully reduce ASR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| relax opacity from zero to 0.2, using FI (text) (ablation) | EIA with SeeAct · Full user request leakage, action step | ASR: 0\% to 16\% | Helped |
| use MI at P_{+1} with GPT-4V | EIA · PII leakage, 177 action steps | ASR: 70\% | Helped |
| inject near the target element at P_{+1} rather than at other studied positions (ablation) | EIA · PII leakage, 177 action steps | ASR averaged across positions: 0.39$\dagger$ | Helped |
| use FI (text) rather than MI (ablation) | EIA on LlavaMistral7B · PII leakage, 177 action steps | ASR mean across positions: 0.05 -> 0.11 | Helped |
| use FI (text) rather than MI (ablation) | EIA on LlavaQwen72B · PII leakage, 177 action steps | ASR mean across positions: 0.18 -> 0.35 | Helped |
| use FI (text) rather than MI (ablation) | EIA on GPT-4V · PII leakage, 177 action steps | ASR mean across positions: 0.38 -> 0.45 | Helped |
| use MI rather than FI (text) (ablation) | EIA on GPT-4V · PII leakage, 177 action steps, P_{+1} | ASR: 0.67 -> 0.70 | Helped |
| use FI (aria) rather than FI (text) (ablation) | EIA on LlavaMistral7B · PII leakage, 177 action steps | ASR mean across positions: 0.11 -> 0.06 | Hurt |
| use FI (aria) rather than FI (text) (ablation) | EIA on LlavaQwen72B · PII leakage, 177 action steps | ASR mean across positions: 0.35 -> 0.22 | Hurt |
| use FI (aria) rather than FI (text) (ablation) | EIA on GPT-4V · PII leakage, 177 action steps | ASR mean across positions: 0.45 -> 0.44 | No clear effect |
| use Relaxed-EIA instead of zero-opacity EIA (ablation) | SeeAct with LlavaMistral7B or LlavaQwen72B · Full user request leakage, action step | ASR: zero | No clear effect |
| add a defensive system prompt warning against prompt injection (ablation) | SeeAct using EIA (MI) or Relaxed-EIA · PII and full user request leakage | ASR | No clear effect |
Surprising findings
- More capable backbones (GPT-4V) were more vulnerable to EIA for PII leakage than weaker open models, which the authors attribute to better instruction following and OCR.
- Standard EIA with zero opacity achieved 0% full-request leakage because action generation only sees the screenshot, which is unaffected by invisible injections.
Limitations
- Offline evaluation on adapted Mind2Web step snapshots rather than end-to-end live web tasks.
- Only one injection instantiation and one target agent (SeeAct) explored; limited exploration of injection variants.
- Full-request leakage only succeeds at low rates (16% max) and only for GPT-4V.
Experimental setup
Datasets: Mind2Web
Benchmarks: Mind2Web
Metrics: attack success rate (ASR) step success rate (SR) ASR_pt ASR_o string similarity threshold 0.95
Baselines: FI (text) FI (aria) MI default SeeAct system prompt VirusTotal
Builds on: SeeAct Mind2Web indirect prompt injection
Closest papers by meaning
- AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents (NeurIPS 2025 · similarity 0.86)
- WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (NeurIPS 2025 · similarity 0.85)
- It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents (ICML 2026 · similarity 0.85)
- VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents (ICLR 2026 · similarity 0.85)
- SafeArena: Evaluating the Safety of Autonomous Web Agents (ICML 2025 · similarity 0.85)
- Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.