Research map / Multi-agent systems and games / Cooperative multi-agent RL
Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large-scale empirical study (82,620 experiments) of how cooperative MARL hyperparameters affect cooperation, robustness, and resilience under 13 uncertainty types across four real-world environments, with hyperparameter tuning shown to improve all three.
Problem
Cooperative MARL policies tuned in ideal simulators often lack robustness to real-world uncertainties and fail to recover (resilience) after disruptions; the relationship between hyperparameters, cooperation, robustness, and resilience is unclear.
Approach
Distinguish robustness (performance under persistent perturbations) from resilience (recovery from perturbed initial states), then systematically vary 15 hyperparameters across MADDPG, MAPPO, and HAPPO to show hyperparameter choices strongly affect trustworthiness metrics, and tuning them alone improves cooperation, robustness and resilience.
Key results
- Hyperparameter tuning (best per task) improves cooperation by 52.60%, robustness by 34.78%, and resilience by 60.34% on average.
- Cooperation-robustness correlation weakens with attack severity (r=0.85 overall, p<.001; cooperation-resilience r=0.76).
- Applying the same tuned hyperparameters to the ERNIE robust method yields average gains of 89.43% cooperation, 65.83% robustness, 82.96% resilience.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use early stopping to select a model by cooperation, robustness, and resilience rather than saving the final training round (ablation) | MARL algorithms · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(161) = 6.16, p<.001 | Helped |
| Use a higher learning rate for the critic than for the actor (ablation) | Actor-critic MARL algorithms · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(161) = 10.02, p<.001 | Helped |
| Replace ReLU with Leaky ReLU (ablation) | MARL algorithms · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(161) = 6.31, p<.001 | Helped |
| Use GAE (ablation) | MAPPO and HAPPO · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(107) = 7.44, p<.001 | Hurt |
| Use PopArt (ablation) | MAPPO and HAPPO · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(107) = 4.84, p<.001 | Hurt |
| Use parameter sharing (ablation) | MARL algorithms · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(161) = 7.01, p<.001 | Hurt |
| Increase exploration using entropy regularization (ablation) | MAPPO and HAPPO · Evaluated real-world environments | combined cooperation, robustness, and resilience: t(107) = 6.64, p<.001 | Helped |
| Increase exploration noise (ablation) | MADDPG · Evaluated real-world environments | combined cooperation, robustness, and resilience | Mixed |
| Optimize cooperation | Cooperative MARL policies · Mild uncertainty | robustness and resilience | Helped |
| Increase uncertainty severity | Cooperative MARL policies · Uncertainty evaluation | correlation of cooperation with robustness: r=0.85, p<.001 | Hurt |
| Train for robustness to one uncertainty modality | MARL policies · Other uncertainty modalities | robustness transfer: F(2, 153 = 9.53, p < .001) | Hurt |
| Train against perturbations applied to all agents | MARL policies · Perturbations applied to individual agents | robustness transfer: F(1, 142) = 4.36, p < .05 | Hurt |
Surprising findings
- Standard practices parameter sharing, GAE, and PopArt reduce cooperation, robustness, and resilience in the benchmark.
- Robustness does not generalize across uncertainty modalities (observation, action, environment) or between single-agent and all-agent perturbations.
Limitations
- Focus restricted to policy-gradient MARL algorithms due to continuous-control requirements.
- Findings reflect general trends and may not hold for every task; correlations in some settings not statistically significant.
Experimental setup
Datasets: DexHand (Bi-DexHands) Quad (quadrotor swarm) Traffic (SUMO ATSC/CACC) Voltage (IEEE 33/141/322-bus)
Benchmarks: DexHand Quad Traffic Voltage
Metrics: cooperative reward robustness (return under uncertainty) resilience (return recovering from perturbed states) Pearson correlation ANOVA paired t-test
Baselines: MADDPG MAPPO HAPPO ERNIE
Builds on: MAPPO HAPPO MADDPG ERNIE SA-MDP PA-AD
Closest papers by meaning
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game (ICLR 2024 · similarity 0.89)
- Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.89)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
- Taming the Curses of Multiagency in Robust Markov Games with Large State Space through Linear Function Approximation (NeurIPS 2026 · similarity 0.88)
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.