Research map / Adversarial robustness and security / Black-box and transfer attacks
AdvAgent: Controllable Blackbox Red-teaming on Web Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdvAgent, a black-box red-teaming framework that trains a Mistral-7B adversarial prompter with SFT then DPO on feedback from SeeAct web agents to inject hidden adversarial prompts into HTML and redirect agent actions.
Problem
Black-box targeted attacks on VLM-based generalist web agents: existing methods require white-box access, manual prompt design, or transfer poorly to black-box agents.
Approach
Inject adversarial prompts into non-rendered HTML attributes (e.g., aria-label) at the expected element, using placeholders for controllable retargeting, and train a generative prompter with SFT on successful prompts followed by DPO using positive and negative black-box agent feedback.
Key results
- AdvAgent achieves 97.5% mean ASR against GPT-4V-based SeeAct, versus 64.3% for InjecAgent and 0% for GCG.
- AdvAgent achieves 99.8% mean ASR against Gemini 1.5-based SeeAct; transfer from GPT-4V gives only 18.0%.
- Under defenses on Gemini 1.5 backend, AdvAgent retains 88.8% to 89.8% mean ASR; SFT-only reaches 69.5% vs 97.5% with DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the prompter with SFT and DPO instead of SFT only (ablation) | AdvAgent · Mind2Web test, all domains (GPT-4V backend) | ASR: 69.5% -> 97.5% | Helped |
| add DPO training after SFT (ablation) | AdvAgent · Mind2Web test, Cooking (D4, GPT-4V backend) | ASR: 49.2% -> 98.0% | Helped |
| replace baseline red-teaming with AdvAgent | SeeAct with GPT-4V · Mind2Web test, all domains | ASR: 64.3 ± 16.7 -> 97.5 ± 2.0 | Helped |
| replace baseline red-teaming with AdvAgent | SeeAct with Gemini 1.5 · Mind2Web test, all domains | ASR: 28.0 ± 23.0 -> 99.8 ± 0.3 | Helped |
| move injection before the expected element instead of after it (ablation) | AdvAgent · Mind2Web test, all domains (GPT-4V backend) | ASR: 71.0 ± 26.1 | Mixed |
| replace aria-label with id as the injection field (ablation) | AdvAgent · Mind2Web test, all domains (GPT-4V backend) | ASR: 97.0 ± 1.7 | No clear effect |
| replace the original target with an alternative target (ablation) | AdvAgent · Mind2Web test, all domains (GPT-4V backend) | ASR: 98.5 ± 2.7 | No clear effect |
| replace the original target with an alternative target (ablation) | AdvAgent · Mind2Web test, all domains (Gemini 1.5 backend) | ASR: 100.0 ± 0.0 | No clear effect |
| transfer successful GPT-4V attacks instead of directly optimizing against Gemini 1.5 (ablation) | AdvAgent · Mind2Web test, all domains (Gemini 1.5 backend) | ASR: 99.8 ± 0.3 -> 18.0 ± 24.4 | Hurt |
| apply random sequence enclosure defense (ablation) | AdvAgent · Mind2Web test, all domains (Gemini 1.5 backend) | ASR: 99.8 -> 89.8 | Hurt |
| apply instruction defense (ablation) | AdvAgent · Mind2Web test, all domains (Gemini 1.5 backend) | ASR: 99.8 -> 88.8 | Hurt |
| apply sandwich defense (ablation) | AdvAgent · Mind2Web test, all domains (Gemini 1.5 backend) | ASR: 99.8 -> 89.8 | Hurt |
Surprising findings
- Transfer-based attacks from GPT-4V-optimized prompts achieve only 18.0% ASR on Gemini 1.5, showing poor cross-backend transfer.
- Changing the injection position reduces ASR in the Finance domain to 26.0%, while changing the HTML field barely matters.
Limitations
- Requires offline victim-agent feedback before prompt optimization; no online adaptation.
- Evaluation is step-based ASR rather than end-to-end task success.
- Evaluated on only one victim framework (SeeAct) and two backends.
Experimental setup
Datasets: Mind2Web
Benchmarks: Mind2Web
Metrics: step-based attack success rate (ASR)
Baselines: GCG Agent-Attack InjecAgent
Builds on: DPO SeeAct GCG InjecAgent Agent-Attack
Closest papers by meaning
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.86)
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · similarity 0.86)
- It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents (ICML 2026 · similarity 0.86)
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.86)
- PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.86)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.