Research map / Multi-agent systems and games / Cooperative multi-agent RL
${\rm E}(3)$-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes E(3)-symmetric cooperative Markov games and builds E(3)-equivariant SEGNN-based actor-critics for MADDPG and MAPPO, improving sample efficiency and zero-shot/transfer generalization on MPE, MuJoCo, and SMAC.
Problem
Cooperative multi-agent RL problems often contain continuous Euclidean (E(3)) symmetries, but prior symmetry-exploiting MARL methods only handle discrete cyclic rotations (C_n, 2D), so continuous 3D symmetries remain unexploited.
Approach
Define group-symmetric Markov games, prove optimal values and invariant observation-based policies exist under the symmetry, and implement actor-critics with steerable E(3)-equivariant message passing networks (SEGNN) over 3D point-cloud representations of states and observations.
Key results
- On MPE, the SEGNN-based critic ([SEGNN, MLP]) outperforms both [GCN, MLP] and [MLP, MLP] baselines in all scenarios, with further gains from a SEGNN actor in Push_N3, Prey_N3, Navigation_N6, Prey_N6.
- SEGNN actors trained on 3-agent MPE tasks achieve near-final performance of baselines at step 0 when zero-shot transferred to 6-agent tasks.
- On SMAC, the SEGNN critic helps in 6h_vs_8z but not in 8m_vs_9m, giving mixed results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a SEGNN critic instead of an MLP or GCN critic | MADDPG on MPE · MPE: Cooperative Navigation, Cooperative Push, Predator and Prey | performance | Helped |
| Add a SEGNN actor to the SEGNN critic (ablation) | MADDPG on MPE · Push_N3, Prey_N3, Navigation_N6, Prey_N6 | performance | Helped |
| Use SEGNN actor-critic instead of MLP actor-critic | MADDPG on MuJoCo · Single- and multi-agent reacher | performance | Helped |
| Use SEGNN actor-critic instead of MLP actor-critic | MADDPG on MuJoCo · Single- and multi-agent swimmer | convergence rate | Helped |
| Use SEGNN actor-critic instead of MLP actor-critic | MADDPG on MuJoCo · Multi-agent 3D hopper and walker | performance | Helped |
| Use a SEGNN critic instead of an MLP critic | MAPPO on SMAC · 6h_vs_8z | performance | Helped |
| Use a SEGNN critic instead of an MLP critic | MAPPO on SMAC · 8m_vs_9m | performance | No clear effect |
| Train a SEGNN-based actor on 3-agent tasks and evaluate it on the corresponding 6-agent tasks | SEGNN-based actor · MPE zero-shot transfer from 3 to 6 agents | performance | Helped |
| Continue training a SEGNN-based actor pretrained on 3-agent tasks in the corresponding 6-agent task | SEGNN-based actor · MPE transfer learning from 3 to 6 agents | convergence rate | Helped |
| Use a SEGNN-based critic instead of a GCN- or MLP-based critic | MADDPG actor on MPE · MPE rotation- and translation-invariance | invariance | Helped |
Surprising findings
- MLP-based actors and critics spontaneously become approximately rotation- and translation-invariant during MADDPG training on Cooperative Navigation, though GCN and MLP critics show no such improvement in the critic-invariance analysis.
- On SMAC, no obvious emergence of invariance occurs, which the authors attribute to competitive scenarios and categorical actions.
Limitations
- Requires knowledge and annotation of strict symmetries in the task, which may not be available.
- Restricted to memory-less, non-recurrent actor-critic architectures even under partial observability.
- SMAC only partially satisfies E(3) symmetry (enemies and categorical actions), limiting gains.
Experimental setup
Datasets: MPE (Cooperative Navigation, Cooperative Push, Predator-and-Prey) MuJoCo continuous control tasks (cartpole, reacher, swimmer, hopper, walker) SMAC (8m_vs_9m, 6h_vs_8z)
Benchmarks: MPE MuJoCo multi-agent tasks SMAC
Metrics: episode return / learning curves sample efficiency zero-shot transfer performance rotation and translation invariance cosine similarity
Baselines: MADDPG [MLP, MLP] MADDPG [GCN, MLP] MAPPO-[MLP, MLP] QMIX
Builds on: MADDPG MAPPO SEGNN MDP homomorphism
Closest papers by meaning
- Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance (NeurIPS 2024 · similarity 0.90)
- Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.87)
- Computing Game Symmetries and Equilibria That Respect Them (AAAI 2025 · similarity 0.87)
- Expected Return Symmetries (ICLR 2025 · similarity 0.87)
- SiT: Symmetry-invariant Transformers for Generalisation in Reinforcement Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.