Research map / Multi-agent systems and games / Cooperative multi-agent RL
Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMU, which combines a trainable conditional autoencoder (dCAE) for semantic episodic memory embeddings with an episodic incentive reward based on state desirability, to speed up and stabilize cooperative MARL on SMAC and Google Research Football.
Problem
Cooperative MARL learns slowly and often gets stuck in local optima on complex tasks; conventional episodic control with random projections yields semantically poor memory recall and can converge to local optima.
Approach
Learn a state embedding (dCAE) that clusters states by their episodic return and reconstructs the state conditioned on timestep, then add an episodic incentive reward r^p = gamma * N_xi/N_call * (H - max Q_target) that is nonzero only for desirable transitions, replacing the fixed-weight episodic-control regularizer.
Key results
- On SMAC super hard maps, EMU (QPLEX) and EMU (CDS) show markedly faster convergence to optimal policy than their original methods and EMC.
- On the hard SMAC map 3s_vs_5z, dCAE embedding reaches an overall win-rate of 0.514 at 2.00M steps with delta=1.3e-7, versus 0.413 for random projection.
- On Google Research Football, EMU quickly finds winning or scoring policies early in training, outperforming QMIX, EMC, QPLEX and CDS without prioritized replay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dCAE embedding instead of random projection (ablation) | EMU without episodic incentive · 3s_vs_5z and 5m_vs_6m | overall win-rate: 0.413 -> 0.514; 0.646 -> 0.677 | Helped |
| Use dCAE embedding instead of EmbNet (ablation) | EMU without episodic incentive · 3s_vs_5z and 5m_vs_6m | overall win-rate: 0.443 -> 0.514; 0.687 -> 0.677 | Mixed |
| Set δ to 1.3e-2 with EmbNet instead of dCAE (ablation) | EMU without episodic incentive · 3s_vs_5z and 5m_vs_6m | overall win-rate: 0.101 vs 0.518; 0.001 vs 0.643 | Hurt |
| Add state reconstruction loss and timestep conditioning (dCAE) rather than EmbNet (ablation) | EMU without episodic incentive · SMAC maps | training efficiency | Helped |
| Remove episodic incentive (ablation) | EMU (QPLEX) and EMU (CDS) · Super-hard SMAC maps | learning performance | Hurt |
| Use conventional episodic control instead of episodic incentive (ablation) | EMU (QPLEX-No-EI) and EMU (CDS-No-EI) · Super-hard SMAC maps | performance variation across seeds | Hurt |
| Use curiosity-based intrinsic reward (ablation) | EMU · Super-hard SMAC except corridor | performance | Hurt |
| Use diversity-based exploration reward (ablation) | EMU · Easy and hard SMAC maps | performance | Hurt |
Surprising findings
- Naive episodic control (EMU without episodic incentive) shows large performance variance across seeds, suggesting it can be detrimental to learning an optimal policy.
- EmbNet, which lacks the reconstruction loss, can select very different states as exploration when they have similar returns, leading to excessive memory recall and failure to find optimal policy at large delta.
Limitations
- Requires choosing a task-dependent threshold delta (Table lists 1.3e-5 for easy/hard SMAC, 1.3e-3 for super hard SMAC and GRF), with performance sensitive to delta on some maps.
- Curiosity and diversity intrinsic rewards were found to hurt performance on some maps, so they were selectively disabled per task.
Experimental setup
Datasets: SMAC Google Research Football
Benchmarks: SMAC Google Research Football
Metrics: test win-rate overall win-rate
Baselines: QMIX QPLEX CDS EMC
Builds on: EMC QPLEX CDS EMDQN PyMARL
Closest papers by meaning
- Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents (NeurIPS 2026 · similarity 0.87)
- Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Sable: a Performant, Efficient and Scalable Sequence Model for MARL (ICML 2025 · similarity 0.86)
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.86)
- A Generalist Hanabi Agent (ICLR 2025 · similarity 0.86)
- Learning to Share: Selective Memory for Efficient Parallel Agentic Systems (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.