Research map / Multi-agent systems and games / Cooperative multi-agent RL

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary

ICML 2025 · Multi-agent systems and games › Cooperative multi-agent RL · approach: Cooperative MARL algorithms

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ATSA, an adversarial training framework for MARL where a stochastic adversary (SDor policy-perturbation director with max-entropy objective, plus STor observation generator) is trained online with the protagonist, plus an SDor-STor loss to align them.

Problem

Robustness of MARL-trained policies to observation perturbations; existing adversarial training overfits to strongest adversarial perturbations, degrading clean performance, and ignores the protagonist's policy information when training the adversary.

Approach

A stochastic adversary built from a maximum-entropy policy director (SDor) that perturbs the protagonist's policy, and an observation generator (STor) guided by SDor via FGSM, linked by a new SDor-STor cross-entropy loss; the authors prove the SDor soft policy converges to a global optimum under factorized max-entropy MARL and yields the optimal observation adversary.

Key results

• On SMAC, ATSA-trained QMIX models achieve average win rate 0.99 on 3m, 0.98 on 3s 3z and 0.99 on 8m across six adversaries, the highest among benchmarks.
• On CAV with VDN, ATSA-trained protagonist averages reward 59.81 and crash rate 0.01, outperforming most baselines; ATSA with QMIX averages 63.74 reward and 0.01 crash rate.
• ATSA is reported to outperform 11 baselines across 3 scenarios with 42 statistically significant wins under Wilcoxon signed-rank tests.

Measured effects reported in the paper

Surprising findings

• Adversaries trained with ATSA do not always produce the strongest attacks; e.g., against QMIX agents, ATSA is not the top attacker, though it is strongest against FGSM(VDN)-trained agents in some settings.

Limitations

• Method is limited to discrete action spaces with deterministic protagonist policies; extension to continuous action spaces is future work
• Stochastic adversary perturbations are not always the strongest attacks
• Perturbation budgets chosen per environment; a extra hyperparameter kappa must be tuned

Experimental setup

Datasets: SMAC (3m, 3s 3z, 8m) CAV (connected autonomous vehicles environment)

Benchmarks: SMAC CAV

Metrics: win rate average reward crash rate Wilcoxon rank-sum significance (p<0.05)

Baselines: NoAdv Random Noise FGSM ATLA PAAD PR PR-REP ERNIE RAP ROMANCE

Builds on: PAAD (policy adversarial actor and director) factorized maximum-entropy MARL FGSM VDN QMIX

Closest papers by meaning

• Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game (ICLR 2024 · similarity 0.89)
• Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.89)
• Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.88)
• Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.87)
• Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · similarity 0.87)
• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.86)