Research map / Multi-agent systems and games / LLM agents
MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Malicious Image Patches (MIPs), adversarially perturbed screen regions via PGD that hijack multimodal OS agents (Windows Agent Arena with Llama-3.2 Vision) into executing malicious API programs when captured in screenshots.
Problem
OS agents that control a computer via VLM-processed screenshots can be hijacked without text access; the paper asks whether a small adversarially perturbed screen patch can force harmful actions across prompts, layouts, and agent components.
Approach
Adversarial perturbations confined to a small screen patch (desktop wallpaper or social media post) are optimised with PGD through the VLM, accounting for screen-parser invariance and image resizing, so that the agent emits a fixed malicious API program; the attack is made universal across prompts, screens, parsers, execution steps and several Llama-3.2 VLMs.
Key results
- Universal MIPs reach 1.00 ASR at greedy decoding on seen prompts/screens and 0.89-1.00 on unseen prompts and screens for Llama-3.2-11B-Vision-Instruct in the desktop setting.
- A single MIP jointly optimised for three Llama-3.2 VLMs achieves about 0.96-1.00 ASR at greedy decoding, but transfers to unseen Llama-3.2-90B-Vision with 0.00 ASR.
- Universal MIPs transfer to an unseen screen parser (GroundingDINO+TesseractOCR), reaching 0.59-0.40 ASR at greedy decoding in the desktop setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize one MIP across prompts and screenshots rather than for a single pair | MIP attack on OS agents · Desktop, unseen prompts and screenshots with OmniParser | ASR: 0.00 -> 1.00 | Helped |
| Optimize one MIP across prompts and screenshots rather than for a single pair | MIP attack on OS agents · Social media, unseen prompts and screenshots, website target, temperature 0.0 | ASR: 0.00 -> 1.00 | Helped |
| Evaluate a targeted MIP on unseen user prompts while keeping its optimized screenshot | Targeted MIP · Desktop, memory overflow target, temperature 0.0 | ASR: 1.00 -> 0.91 | Helped |
| Evaluate a targeted MIP on unseen screenshots | Targeted MIP · Desktop, memory overflow target, temperature 0.0 | ASR: 1.00 -> 0.00 | Hurt |
| Replace OmniParser with GroundingDINO and TesseractOCR | Universal MIP · Desktop, unseen prompts and screenshots, memory overflow target, temperature 0.0 | ASR: 1.00 -> 0.59 | Hurt |
| Capture the universal MIP after benign-task execution rather than at the initial step | Universal MIP · Social media screenshots, memory overflow target, temperature 0.0 | ASR: 1.00; 0.67 | Mixed |
| Jointly optimize the MIP across multiple VLMs | Universal MIP · Desktop, seen and unseen prompts and screenshots, optimized VLMs | ASR: 1.00; 0.96 | Helped |
| Transfer a jointly optimized MIP to an unseen Llama-3.2-90B-Vision VLM | Jointly optimized MIP · Desktop, all reported prompt/screenshot combinations | ASR: 1.00 -> 0.00 | Hurt |
Surprising findings
- MIPs fail completely (0.00 ASR) on an unseen VLM from the same family (Llama-3.2-90B-Vision), indicating poor cross-model transfer.
- Targeted MIPs optimised for one screenshot do not transfer to unseen screenshots (ASR 0.00).
Limitations
- Transfer to unseen VLMs fails, so attacks require knowledge of the agent's VLM (grey-box).
- Requires the MIP to be captured by an agent using one of the optimised VLMs and the patch to be encountered.
- Evaluated only on Windows Agent Arena with two malicious behaviours and a fixed default system prompt.
Experimental setup
Datasets: Windows Agent Arena tasks (user prompts) DALL·E 3 generated desktop background Bluesky social media post screenshots
Benchmarks: Windows Agent Arena
Metrics: average success rate (attack success rate over multinomial s
Baselines: OmniParser parser with GroundingDINO and TesseractOCR (WAA baseline parser)
Builds on: projected gradient descent Set-of-Marks OmniParser Windows Agent Arena
Closest papers by meaning
- Seeing Is Not Screening: Multimodal Hidden Instruction Attacks on Agent Skill Scanners (NeurIPS 2026 · similarity 0.83)
- macOSWorld: A Multilingual Interactive Benchmark for GUI Agents (NeurIPS 2025 · similarity 0.83)
- GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.82)
- AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.82)
- Image Hijacks: Adversarial Images can Control Generative Models at Runtime (ICML 2024 · similarity 0.82)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.