Research map / Multi-agent systems and games / LLM agents

MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaMa, a Stackelberg-game-based automated design method where an LLM Meta-Agent iteratively designs multi-agent systems defended against worst-case attacks found by an LLM Meta-Adversary that compromises agents.

Problem

Automated design of LLM-based multi-agent systems is optimized for task performance and ignores safety under agent compromise (failures or adversarial agents).

Approach

Formalize safe system design as a Stackelberg security game: the Meta-Agent (leader) proposes systems and a best-responding Meta-Adversary searches for worst-case compromises of a subset of agents; the Meta-Agent updates designs using the top-k discovered attacks as feedback, extending AFlow with a structured agent/tool/communication-graph representation.

Key results

• Meta-Adversary reduces safety of initial systems with LPTG+TF defenses from ~4.3-4.9 (clean) to 1.5-3.2 under top-5 attacks across four environments.
• On targeted attacks, MaMa achieves ASR of 0.17 (Travel Planning) and 0.15 (Personal Assistant) versus 0.52-0.56 and 0.31-0.39 for LPTG+TF and Guardian Agents baselines.
• Under indirect tool injection, MaMa achieves ASR 0.0068 on Travel Planner vs 0.1020 for Guardian Agents; it is not best on Code Generation (0.0968 vs 0.0787 for Delimiters).

Surprising findings

• Designing for more adversaries (2-3) yields safer systems but at a notable quality cost, e.g. Personal Assistant quality drops to 1.5 with 3 adversaries.
• Stronger guard-style defenses, including Guardian Agents, offer only modest gains over baseline guardrails in some environments.

Limitations

• Computationally expensive: runtimes of 2-5 days per environment for 25 generations.
• Adversary only minimizes safety, ignoring quality; threat model restricted to compromising agents without changing tools or communication graph.
• Evaluated on only a few environments with open-source agent models and LLM judges; results may not transfer to real-world systems.

Experimental setup

Datasets: BAD-ACTS AgentDojo

Benchmarks: BAD-ACTS AgentDojo

Metrics: safety score (1-5, judge) quality score (1-5, judge) attack success rate (ASR) quadratic weighted Cohen's kappa

Baselines: Least-Privilege Tool Gating + Tool Filters Guardian Agents Delimiters quality-only MaMa (AFlow-style)

Builds on: AFlow Stackelberg security games BAD-ACTS AgentDojo

Closest papers by meaning

• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.89)
• SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.88)
• ResMAS: Resilience Optimization in LLM-based Multi-agent Systems (AAAI 2026 · similarity 0.87)
• When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems (ICML 2026 · similarity 0.86)
• Agent Security is a Systems Problem (NeurIPS 2026 · similarity 0.86)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.86)