Research map / Multi-agent systems and games / Cooperative multi-agent RL
LAGMA: LAtent Goal-guided Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAGMA, which learns a quantized latent state space with a modified VQ-VAE (coverage loss), stores top-k goal-reaching latent trajectories, and adds a latent goal-guided intrinsic reward to the TD target in cooperative MARL.
Problem
Cooperative MARL agents learn slowly and often fail to reach semantic goals (e.g., defeating all enemies, scoring a goal) in complex dense or sparse reward tasks, and goal-conditioned RL ideas do not transfer easily to MARL.
Approach
Discretize global states with a VQ-VAE whose coverage loss spreads codebook vectors over the feasible latent space, keep top-k high-return latent trajectories per code in an extended codebook, and add a non-negative intrinsic reward for transitions onto a sampled goal-reaching path, which the authors argue yields a more accurate TD target.
Key results
- On SMAC MMM2 policy robustness test, LAGMA reaches 0.975 nominal win rate versus 0.381 for QMIX and 0.444 for LDSA (trained 1M steps).
- Extended codebook memory overhead estimated at about 2.19 MiB maximum for D_tau and 25.6 KiB for D_VQ in the example configuration.
- On SMACv2 LAGMA shows only comparable or better performance, not distinctively strong, which the authors attribute to lack of diverse reference trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LAGMA instead of QMIX | SMAC MMM2 policy trained on nominal map · SMAC MMM2 NW(hard), NW, Nominal, SW, and SW(hard) | win rate: 0.050 ± 0.036 -> 0.275 ± 0.064; 0.138 ± 0.100 -> 0.500 ± 0.104; 0.381 ± 0.078 -> 0.975 ± 0.026; 0.194 ± 0.092 -> 0.556 ± | Helped |
| use LAGMA instead of LDSA | SMAC MMM2 policy trained on nominal map · SMAC MMM2 NW(hard), NW, Nominal, SW, and SW(hard) | win rate: 0.000 ± 0.000 -> 0.275 ± 0.064; 0.081 ± 0.047 -> 0.500 ± 0.104; 0.444 ± 0.107 -> 0.975 ± 0.026; 0.063 ± 0.049 -> 0.556 ± | Helped |
| remove coverage loss (ablation) | LAGMA · SMAC 3m and 2s3z (sparse) | performance | Hurt |
| replace timestep-dependent coverage loss with coverage loss over all codebook vectors at each timestep (ablation) | LAGMA · SMAC 3m and 2s3z (sparse) | performance | Hurt |
| use Cq0 instead of Cq,t for trajectory value estimation (ablation) | LAGMA · SMAC 3m and 2s3z (sparse) | performance stability | Hurt |
| generate the goal-reaching trajectory only at the initial state (ablation) | LAGMA · SMAC 3m and 2s3z (sparse) | performance stability | Hurt |
Surprising findings
- On SMACv2, LAGMA does not show distinctively strong performance, unlike on other benchmarks.
Limitations
- Performance on SMACv2 with diverse goals is not distinctive because only return-based storage of reference trajectories is used.
- Adds hyperparameters (codebook size, update intervals, coverage loss weight) that require per-task tuning.
- Training time is higher than some baselines on MMM2 (17.7h vs 9.8h LDSA).
Experimental setup
Datasets: SMAC Google Research Football
Benchmarks: SMAC Google Research Football SMACv2 27m_vs_30m
Metrics: win rate training time memory usage
Baselines: QMIX RODE LDSA MASER EMC
Builds on: QMIX VQ-VAE EMC MASER
Closest papers by meaning
- CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (ICLR 2020 · similarity 0.88)
- Multiagent Q-learning with Sub-Team Coordination (NeurIPS 2022 · similarity 0.86)
- Trajectory-Class-Aware Multi-Agent Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Learning to Collaborate with Unknown Agents in the Absence of Reward (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.