Research map / Multi-agent systems and games / Cooperative multi-agent RL
Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E2GN2, an exploration-enhanced equivariant GNN policy for MARL that removes the early exploration bias of EGNN while keeping O(n) equivariance, evaluated with PPO on MPE and SMACv2.
Problem
MARL is sample-inefficient and generalizes poorly because standard MLP/GNN policies lack symmetry inductive bias; naive EGNN policies show a biased early exploration (actions centered on agent position) and mapping continuous equivariant outputs to discrete action logits is non-trivial.
Approach
Adds a learned invariant scalar gate phi_u2(m_i) multiplying the node's own coordinate embedding in the EGNN coordinate update, so the expected initial action mean is zero (unbiased exploration) while O(n) equivariance is retained; also maps invariant features to discrete actions and equivariant coordinates to movement actions to handle mixed action spaces.
Key results
- On SMACv2 terran, E2GN2 reaches 0.4 win rate after about 1.25x10^6 timesteps versus 5x10^6 for GNN and MLP never exceeding ~0.3 by 10x10^6.
- In the Surrounded Right generalization test (trained on Surrounded Left), E2GN2 achieves 0.55 win rate on terran versus 0.11 for GNN and 0.12 for MLP, roughly 5x the baselines.
- E2GN2 keeps its win rate under the Surrounded Left to Right shift (0.57 to 0.55 on terran, 0.59 to 0.56 on protoss), while GNN and MLP drop substantially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E2GN2 instead of GNN | SMACv2 policies trained on Surrounded Left · SMACv2 Terran, Surrounded Right | win rate: 0.11 -> 0.55 | Helped |
| Use E2GN2 instead of MLP | SMACv2 policies trained on Surrounded Left · SMACv2 Protoss, Surrounded Right | win rate: 0.17 -> 0.56 | Helped |
| Use E2GN2 instead of GNN | SMACv2 policies · SMACv2 Zerg, Surrounded Right | win rate: 0.07 -> 0.32 | Helped |
| Scale the trained policy from 5 to 8 agents | E2GN2 · SMACv2 Terran | win rate: 0.69 -> 0.54 | Hurt |
| Scale the trained policy from 5 to 8 agents | GNN · SMACv2 Terran | win rate: 0.45 -> 0.32 | Hurt |
| Use E2GN2 instead of EGNN to mitigate early exploration bias | PPO agents · MPE tag | final reward | Helped |
| Use E2GN2 instead of standard GNNs and MLPs | PPO agents · SMACv2 Terran | timesteps to 0.4 win rate: 1.25×10^6 timesteps; 5×10^6 timesteps | Helped |
| Train EGNN without recentering observations (ablation) | EGNN · SMACv2, Surrounded, partial observability | performance | Hurt |
| Recenter observations (ablation) | E2GN2 · SMACv2, Surrounded, partial observability | performance | No clear effect |
Surprising findings
- Naive EGNN policies move agents away from goals early in training, giving low initial reward, although EGNN still eventually outperforms GNN and MLP on MPE.
- E2GN2 performed well under partial observability and with the centering step removed, while EGNN degraded without observation centering.
Limitations
- E2GN2 is not translation equivariant, which the authors argue is acceptable for MARL but limits the symmetry group.
- Main experiments assume full observability; partial observability is only covered in supplementary results.
- Theoretical unbiasedness relies on approximate expectation assumptions (independence, zero-mean initialization) and single-layer analysis.
Experimental setup
Datasets: MPE simple spread MPE tag SMACv2 terran SMACv2 protoss SMACv2 zerg
Benchmarks: MPE simple spread MPE tag SMACv2
Metrics: reward win rate sample efficiency (timesteps to reach win rate)
Baselines: MLP GNN EGNN E3AC
Builds on: EGNN PPO RLlib SMACv2 MPE
Closest papers by meaning
- ${\rm E}(3)$-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.90)
- SiT: Symmetry-invariant Transformers for Generalisation in Reinforcement Learning (ICML 2024 · similarity 0.87)
- HGAP: Boosting Permutation Invariant and Permutation Equivariant in Multi-Agent Reinforcement Learning via Graph Attention Network (ICML 2024 · similarity 0.87)
- Learning (Approximately) Equivariant Networks via Constrained Optimization (NeurIPS 2025 · similarity 0.87)
- Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- A Practical Guide for Incorporating Symmetry in Diffusion Policy (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.