Research map / Multi-agent systems and games / LLM agents

When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AcMAS, which detects malicious agents in LLM multi-agent systems by measuring cosine divergence of final-layer activations from a benign centroid, and restores compromised agents by activation steering instead of isolating them.

Problem

Existing MAS security defenses rely on semantically explicit attacks and graph-based topology models that break under stealthy attacks and asynchronous execution, and their isolation-based mitigations disrupt task completion.

Approach

Model normal agent reasoning as a unit-normalized centroid in the LLM's final-layer activation space, flag agents by cosine divergence, and correct flagged agents by steering their activations toward that centroid, with no interaction graph required.

Key results

• On synchronous MAS, AcMAS reaches F1 0.93-0.95 across five scenarios versus 0.70-0.74 for G-Safeguard on the stealthy attacks, per the abstract.
• In asynchronous MAS, AcMAS F1 is 0.92-0.94 while G-Safeguard drops to about 0.38-0.46 (abstract cites 0.93 vs 0.38).
• Correction preserves task completion (avg TCR 0.98) versus about 0.83 for isolation-based G-Safeguard, and reduces ASR to about 0.04.

Surprising findings

• The paper reports that the isolation-based baseline keeps detection accuracy similar to the full method in ablation, but task completion falls to 0.68 when correction is replaced with isolation.

Limitations

• Requires access to internal activations, so it applies only to open-weight LLM agents.
• Performance degrades under domain shift until target-domain benign traces are added; adaptive attacks that gradually shift activations are not evaluated.
• Results are for homogeneous MAS only; mixed-backbone MAS is left to future work.

Experimental setup

Datasets: CSQA GSM8K InjecAgent PoisonRAG HotPotQA

Benchmarks: CSQA GSM8K InjecAgent PoisonRAG HotPotQA

Metrics: F1 precision recall FPR AUROC TCR ASR

Baselines: G-Safeguard BlindGuard PERM TAM

Builds on: G-Safeguard BlindGuard PERM TAM

Closest papers by meaning

• Securing Multi-Agent Systems Against Corruptions via Node Contribution Backpropagation (ICML 2026 · similarity 0.89)
• Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAS (AAAI 2026 · similarity 0.88)
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.87)
• When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems (ICML 2026 · similarity 0.87)
• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.87)
• AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks (ICML 2026 · similarity 0.87)