Research map / Multi-agent systems and games / LLM agents

Breaking and Fixing Defenses Against Control Flow Hijacking in Multi-Agent Systems

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows control-flow hijacking attacks evade LlamaFirewall alignment checks and proposes ControlValve, which enforces LLM-generated control-flow graphs plus edge-specific contextual rules at the orchestrator of multi-agent systems.

Problem

Multi-agent systems with black-box agents are vulnerable to control-flow hijacking, where injected fake errors redirect the orchestrator into invoking unsafe agents; existing alignment-check defenses (e.g., LlamaFirewall) can be evaded.

Approach

Apply control-flow integrity and least privilege at the orchestration layer: generate a task-specific context-free grammar of permitted agent-call sequences and zero-shot, per-edge natural-language contextual rules, and enforce them with narrow checks rather than open-ended alignment judgments.

Key results

• ControlValve reduces attack success rate to 0% across all evaluated IPI and CFH attacks in CFH-Hard coding, computer use, and AgentDojo Slack, while LlamaFirewall (default Llama backing) still allows up to 80% of CFH-Hard coding attacks and 67% computer-use attacks.
• ControlValve achieves 97% single-answer accuracy on CFH-Hard coding and 100% on computer use, versus 93% and 89% for the undefended base.
• On AgentDojo Slack, ControlValve gets 62% benign task success vs 65% for the undefended base, with runtime of 128 s vs 51 s for undefended.

Surprising findings

• Undefended multi-agent systems often continue searching and execute adversarial code after refusing it, due to conflict between helpfulness and safety.
• Vague non-adversarial inputs cause accidental sensitive-data cc violations in 56% of baseline runs.

Limitations

• LLM-generated control-flow graphs and rules can be too permissive or too restrictive, causing failures or hallucinated tools/URLs.
• Runtime overhead may be intractable in some settings.
• Unclear whether evasion attacks compliant with the CFG are possible in multi-agent systems.

Experimental setup

Datasets: CFH-Hard AgentDojo Triedman et al. CFH payloads

Benchmarks: CFH-Hard AgentDojo Slack

Metrics: attack success rate benign task success rate open-ended win-or-tie rate runtime

Baselines: Undefended MAS Least privilege LlamaFirewall Azure Content Filters

Builds on: LlamaFirewall Magentic-One AgentDojo AutoGen Control-flow hijacking (Triedman et al.)

Closest papers by meaning

• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.87)
• The Best-Laid SCHEMEs: Coordinated Sabotage and Monitoring in Multi-Agent Systems (NeurIPS 2026 · similarity 0.87)
• AI Agents May Always Fall for Prompt Injections (NeurIPS 2026 · similarity 0.85)
• Architecture Matters for Multi-Agent Security (ICML 2026 · similarity 0.85)
• Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems (AAAI 2026 · similarity 0.84)
• AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks (ICML 2026 · similarity 0.84)