Research map / Multi-agent systems and games / Cooperative multi-agent RL
Expected Return Symmetries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces expected return symmetries (ER symmetries), a broader symmetry group than Dec-POMDP symmetries, learned from interaction via gradient-based search, and uses them in other-play to improve zero-shot coordination in Hanabi, Overcooked V2 and toy games.
Problem
Zero-shot coordination in cooperative Dec-POMDPs suffers from mutually incompatible symmetry breaking; existing other-play requires a priori known environment symmetries, which are hard to discover automatically in large Dec-POMDPs.
Approach
Define a symmetry group of action-observation relabelings that preserves the self-play optimal set of Boltzmann-exploratory policies (rather than the environment dynamics), which strictly contains Dec-POMDP symmetries, and learn approximate elements of it with policy-gradient optimization over observation permutations plus fixed action transpositions.
Key results
- In Hanabi, IPPO + OP with ER symmetries reaches mean XP 21.64 vs 8.61 for OP with Dec-POMDP symmetries and 4.02 for IPPO.
- In Overcooked V2, the ER symmetry population has mean XP 15.8 versus 6.74 for the IPPO self-play baseline.
- The toy coordination game learner recovers the action swap, which is an ER symmetry but not a Dec-POMDP symmetry.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OP with learned expected return symmetries instead of self-play | IPPO population · Hanabi | XP: 4.02 ± 0.17 -> 21.64 ± 0.07 | Helped |
| Use OP with learned expected return symmetries instead of OP over Dec-POMDP symmetries | IPPO population · Hanabi | XP: 8.61 ± 0.17 -> 21.64 ± 0.07 | Helped |
| Use OP with learned expected return symmetries instead of self-play | IPPO population · Overcooked V2, Grounded Coordination Simple | mean XP: 6.74 -> 15.8 | Helped |
| Enforce compositionality and invertibility in ER symmetry learning (ablation) | ER symmetry learning · Hanabi, 6 unseen optimal policies | Rel. Rec. Loss: 31.4% ± 1.8% -> 16.7% ± 0.18% | Helped |
| Enforce compositionality and invertibility in ER symmetry learning (ablation) | ER symmetry learning · Hanabi, 6 unseen optimal policies | Single Transform: 22.88 ± 0.07 -> 23.32 ± 0.05 | Helped |
| Enforce compositionality and invertibility in ER symmetry learning (ablation) | ER symmetry learning · Hanabi, 6 unseen optimal policies | Double Comp.: 21.10 ± 0.09 -> 22.16 ± 0.07 | Helped |
| Enforce compositionality and invertibility in ER symmetry learning (ablation) | ER symmetry learning · Hanabi, 6 unseen optimal policies | Triple Comp.: 20.36 ± 0.11 -> 20.94 ± 0.12 | Helped |
| Apply the ER symmetrizer at test time | Hanabi IPPO + OP^ER population · Hanabi | XP(*): 22.03 ± 0.05 -> 22.50 ± 0.06 | Helped |
| Apply the ER symmetrizer at test time | Hanabi IPPO + OP^MDP population · Hanabi | XP(*): 8.14 ± 0.15 -> 9.91 ± 0.14 | Helped |
| Apply the MDP symmetrizer at test time | Hanabi IPPO population · Hanabi | XP(*): 0.12 ± 0.03 -> 0.14 ± 0.03 | No clear effect |
| Apply the ER symmetrizer at test time | Hanabi IPPO population · Hanabi | XP(*): 0.12 ± 0.03 -> 0.10 ± 0.03 | No clear effect |
| Use OP over Dec-POMDP symmetries in the two-lever variant | OP^MDP · Two-lever game | zero-shot coordination | Hurt |
Surprising findings
- Symmetrization with Dec-POMDP symmetries improved the IPPO baseline's cross-play only marginally, while the ER-OP population outperforms the Dec-POMDP-OP population even though the latter uses all 120 environment symmetries.
- The three-lever game shows OP with Dec-POMDP symmetries can fail in the two-lever variant because the symmetry group lacks expressivity.
Limitations
- Symmetry search is restricted to bijections over action and observation spaces, limiting expressivity (e.g., fails in the two-lever game).
- Success depends heavily on the type of policies used to learn ER symmetries.
- Learned symmetries only approximately satisfy group properties and expected-return preservation.
Experimental setup
Datasets: Hanabi Overcooked V2 Iterated three-lever game Toy coordination game
Benchmarks: Hanabi Overcooked V2 (Grounded Coordination Simple) Iterated three-lever game Toy coordination game
Metrics: self-play score cross-play score median cross-play score expected return
Baselines: IPPO self-play IPPO + OP with Dec-POMDP symmetries off-belief learning cognitive hierarchy
Builds on: other-play zero-shot coordination off-belief learning IPPO JaxMARL
Closest papers by meaning
- ${\rm E}(3)$-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- Latent Mixture of Symmetries for Sample-Efficient Dynamic Learning (NeurIPS 2025 · similarity 0.86)
- Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.86)
- SymDrift: One-Shot Generative Modeling under Symmetries (NeurIPS 2026 · similarity 0.85)
- Remove Symmetries to Control Model Expressivity and Improve Optimization (ICLR 2025 · similarity 0.85)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.