Research map / Multi-agent systems and games / LLM agents

AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentHijack, a benchmark of 9 configurable common environment corruptions applied to OSWorld (3,321 tasks), and AgentHijack-Agent, an RL-trained (DA-GRPO) UI-TARS-1.5-7B agent with an onlooker module for behavior summarization and environment checking.

Problem

Computer-use agents built on MLLMs are fragile to common, non-adversarial environment corruptions (pop-ups, resolution changes, network errors, login prompts), yet existing benchmarks mostly measure clean-environment success or adversarial/prompt-injection robustness.

Approach

A configurable corruption suite injected into executable OSWorld VMs to measure corruption robustness, plus a framework that pairs an RL-tuned action generator (DA-GRPO, rollouts across corrupted environments) with an onlooker agent that summarizes environment changes and checks initial environment state.

Key results

• UI-TARS-1.5-7B averages 18.74% success under corruptions vs 24.21% on clean tasks.
• AgentHijack-Agent improves average success to 22.89% (+4.15 points over UI-TARS-1.5-7B) and clean success to 27.80%.
• Pop-up corruption drops UI-TARS-1.5-7B to 10.28%; AgentHijack-Agent reaches 21.51% under pop-ups.

Surprising findings

• Gemini-2.5-Pro and GPT-4o score very low under corruptions (average 5.82% and 3.69%), with the open UI-TARS models far outperforming general MLLMs.
• Resolution change remains a weak spot: AgentHijack-Agent only gains +0.84 points under resolution change (12.53%).

Limitations

• Improvement over the base model is modest (about 4 points average).
• Training uses only 128 tasks, and the onlooker default is the same fine-tuned UI-TARS-1.5-7B model.

Experimental setup

Datasets: AgentHijack (OSWorld-based corrupted tasks) OSWorld

Benchmarks: AgentHijack

Metrics: task success rate

Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.5-Pro Qwen2.5-VL-72B-Instruct Llama-3.2-90B-Vision-Instruct GLM-4.5V UI-TARS-7B-DPO UI-TARS-72B-DPO UI-TARS-1.5-7B

Builds on: GRPO ARPO UI-TARS OSWorld VERL

Closest papers by meaning

• Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.90)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.90)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.89)
• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.89)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.89)