Research map / Adversarial robustness and security / Attacks on text and graphs
RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
Zeyi Liao, Jaylen Jones, Linxi Jiang, Yuting Ning, Eric Fosler-Lussier, Yu Su, Zhiqiang Lin, Huan Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Operator with built-in confirmation and safety checks instead of Claude 3.7 Sonnet CUA / GPT-4o scaffolding | Operator · RTC-Bench Decoupled | ASR (%): -58.62% ASR | Improved |
| Disable Operator's confirmation and safety checks (inattentive supervision) (ablation) | Operator · RTC-Bench Decoupled (avg) | ASR (%): 7.57 -> 30.89 | Hurt |
| Use Claude 3.7 Sonnet CUA specialization instead of base Claude 3.7 Sonnet | Claude 3.7 Sonnet · RTC-Bench Decoupled (avg) | ASR (%): 39.33 -> 42.93 | Hurt |
| Use Claude 3.5 Sonnet CUA specialization instead of base Claude 3.5 Sonnet | Claude 3.5 Sonnet · RTC-Bench Decoupled (avg) | ASR (%): 41.37 -> 31.21 | Improved |
| Evaluate End2End instead of Decoupled Eval (agent must navigate to injection itself) | Operator (w/o checks) · RTC-Bench 50 high-risk tasks | ASR (%): 94.0 -> 42.0 | Improved |
| Use more capable Claude 4.5 Opus CUA instead of Claude 3.7 Sonnet CUA (End2End) | Claude CUAs · RTC-Bench End2End (50 tasks) | ASR (%): 50 -> 83 | Hurt |
| Upgrade Claude Opus 4.5 CUA to Claude Opus 4.6 CUA | Claude Opus CUA · RTC-Bench End2End (50 tasks) | ASR (%): 83 -> 50 | Improved |
| Add defensive system prompt instructing injection detection (ablation) | Claude 3.7 Sonnet CUA and Operator (w/o checks) · RTC-Bench 50 high-risk tasks | ASR (%): ASR remain near 50% | No clear effect |
| Add LlamaFirewall or PromptArmor detectors (ablation) | CUAs · RTC-Bench 50 high-risk tasks | injection detection rate: best variant detects only 30% | No clear effect |
| Use Meta SecAlign model-level defense (ablation) | Meta SecAlign · RTC-Bench 50 high-risk tasks | ASR: follows malicious instructions in about half of tasks | No clear effect |
| Move from Confidentiality to Integrity goals for Claude 3.5 Sonnet on OwnCloud | Claude 3.5 Sonnet · OwnCloud | ASR (%): 0.00 -> 48.67 | Improved |
Closest papers by meaning
- VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents (ICLR 2026 · similarity 0.87)
- AdvAgent: Controllable Blackbox Red-teaming on Web Agents (ICML 2025 · similarity 0.87)
- ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs (NeurIPS 2026 · similarity 0.87)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.86)
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.86)
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
