Research map / Multi-agent systems and games / Cooperative multi-agent RL

Learning Joint Behaviors with Large Variations

AAAI 2025 · Multi-agent systems and games › Cooperative multi-agent RL · approach: Intrinsic-reward exploration for MARL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DJBD, a multi-agent exploration method that learns a 1-Lipschitz representation function to maximize traveled distances in joint behavior space, implemented on top of QMIX (and MAPPO), evaluated on LBF, SMAC, and SMACv2.

Problem

Insufficient exploration in cooperative MARL; mutual information-based exploration methods can be maximized with only small differences between joint behaviors, so agents do not learn joint behaviors with large variations.

Approach

Replace mutual information with a conditional mutual information bound whose direction term aligns latent-space joint behavior differences with latent variables; enforce a 1-Lipschitz constraint (spectral normalization) on the representation so that larger latent differences correspond to larger traveled distances in joint behavior space, used as a per-step transition reward in QMIX.

Key results

• DJBD outperforms QMIX and mutual information baselines (MAVEN, EOI, CDS, FoX, etc.) across SMAC scenarios including 6h vs 8z and corridor (super hard).
• On SMACv2 terran, protoss, and zerg 5 vs 5, DJBD achieves significant outperformance over baselines in all scenarios.
• Ablating the 1-Lipschitz constraint drops performance to roughly QMIX level, showing the constraint is essential.

Measured effects reported in the paper

Surprising findings

• Maximizing the direction objective without the Lipschitz constraint yields performance similar to QMIX, so enlarging representation-space distance alone does not enlarge joint behavior distance.
• The method does not improve over baselines in simple scenarios such as 3s5z (easy) and 2c vs 64zg (hard), suggesting diversity may not help learning efficiency there.

Limitations

• Gains are limited or absent in simple scenarios where diversity does not aid learning.
• Relies on a Lipschitz-constrained representation, and the 1-Lipschitz constraint is critical; variants without it fail.

Experimental setup

Datasets: LBF SMAC SMACv2

Benchmarks: LBF SMAC SMACv2

Metrics: win rate average return average normalized joint behavior distance food items eaten visitation heatmap coverage

Baselines: QMIX QTRAN MAVEN EOI CDS PMIC LIPO FoX

Builds on: QMIX MAVEN MAPPO spectral normalization

Closest papers by meaning

• Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.90)
• Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.89)
• Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• O-MAPL: Offline Multi-agent Preference Learning (ICML 2025 · similarity 0.88)
• Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.88)
• Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.88)