Research map / Multi-agent systems and games / LLM agents

AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdapAction, a backdoor attack that implants a context-aware adaptive malicious action policy into MLLM GUI agents via SFT on teacher-generated malicious reasoning trajectories, achieving up to 100% ASR while evading a multi-principle LLM-based defense.

Problem

Existing backdoor attacks on GUI agents map triggers to fixed, context-agnostic malicious actions, which are semantically incoherent and easily detected; the paper targets stealthy, context-adaptive backdoors that preserve utility.

Approach

Instead of a fixed trigger-action mapping, the backdoored agent is trained (via SFT on structured malicious reasoning trajectories from a strong adversarial teacher MLLM) to select a contextually coherent malicious action from a five-category library based on the GUI state, instruction, and trigger.

Key results

• AdapAction reaches 100% ASR on AitZ for both Qwen2-VL-2B and 7B, and 100% on AndroidControl with 7B (96.55% with 2B).
• Under a multi-principle defense, AdapAction gets 0% DR@3 and 38% DR@1 versus over 94% DR@1 for fixed-target BadNets/ICLAttack, close to the 28% clean false-positive rate.
• Utility stays close to the clean model, e.g., 75.93% AMR vs 76.10% clean on AndroidControl 7B.

Surprising findings

• Text-only teacher (LLaMA-3-70B) yields much higher detectability (63% DR@1) than multimodal teachers, indicating visual grounding of teacher matters.

Limitations

• Malicious action library is predefined with only five categories, limiting adaptiveness to that set.
• Evaluation of the defense is a single LLM-based multi-principle framework; other defenses are only in supplementary material.
• Attack requires control over SFT data (supply-chain threat model), not applicable to inference-only settings.

Experimental setup

Datasets: AitZ AndroidControl

Benchmarks: AitZ AndroidControl AndroidWorld

Metrics: ASR AMR TMR DR@1 DR@3 MCD

Baselines: BadNets ICLAttack AgentGhost Clean model

Builds on: AgentGhost VIBMA BadNets ICLAttack

Closest papers by meaning

• D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies (AAAI 2026 · similarity 0.87)
• DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections (NeurIPS 2026 · similarity 0.87)
• LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments (NeurIPS 2026 · similarity 0.86)
• GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.86)
• Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.86)
• macOSWorld: A Multilingual Interactive Benchmark for GUI Agents (NeurIPS 2025 · similarity 0.85)