Research map / Multi-agent systems and games / LLM agents

AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdvEvo-MARL, a co-evolutionary MARL framework where attacker and defender LLM agents are trained jointly with a group-level public mean-return baseline, internalizing safety into task agents without external guards.

Problem

Multi-agent LLM systems are vulnerable to jailbreaks, prompt injection, and adversarial collaboration; existing defenses rely on self-verification (weak) or external guard agents (single point of failure and overhead).

Approach

Jointly trains attacker and defender LLMs under adversarial MARL (after SFT warm-up of attackers), with a public baseline in which agents in the same functional group share the group mean return as advantage baseline, and defenders receive both local and global safety/task rewards.

Key results

• AdvEvo-MARL keeps ASR below 20% across scenarios, versus baselines reaching up to 38.33% ASR (UserHijack, Challenger-7B).
• AdvEvo-MARL-7B achieves 0% ASR on AutoInject chain and tree settings and 0.57% on GPQA chain, while task accuracy rises up to about +3.67% on reasoning tasks.
• Only a maximum ~3% accuracy drop for 3B variants on task benchmarks; 7B variant trained only on math outperforms vanilla on all three task datasets.

Surprising findings

• Challenger-7B (self-verification) increases ASR by about 10 points over its undefended vanilla variant under UserHijack.
• Standard (no public baseline) training shows a ~13.3% drop in defender response length over the last 50 steps, interpreted as defensive overcompensation.

Limitations

• Experiments limited to three attack scenarios and three MAS topologies with three agents; attacker/defender reward design and LLM judge details are sparsely specified.
• Some ASR/CR values are not consistently better than baselines (e.g., AdvEvo-MARL-3B CR 35.64% on chain NetSafe, and complete topology AutoInject ASR above baselines), and the paper's claims of 'consistent' gains are mixed.
• Compute, training hyperparameters, and judge model are not clearly reported in the main text.

Experimental setup

Datasets: MATH-500 JailbreakBench WildJailbreak StrongREJECT NetSafe evaluation set

Benchmarks: NetSafe AutoInject UserHijack AIME'24 AIME'25 LiveCodeBench v6 GPQA-Diamond

Metrics: attack success rate (ASR) contagion rate (CR) accuracy Pass@1

Baselines: Vanilla Qwen2.5-3B Vanilla Qwen2.5-7B Challenger Inspector GPT-3.5 GPT-4o-mini

Builds on: REINFORCE++ GRPO NetSafe AutoInject

Closest papers by meaning

• MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems (ICML 2026 · similarity 0.89)
• Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.88)
• ResMAS: Resilience Optimization in LLM-based Multi-agent Systems (AAAI 2026 · similarity 0.88)
• Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.88)
• Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents (ICML 2026 · similarity 0.88)
• SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.87)