Research map / Multi-agent systems and games / Cooperative multi-agent RL
Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MADPO, a sequential-updating MARL method that maximizes mutual (intra- and inter-agent) policy divergence using a conditional Cauchy-Schwarz divergence for entropy-guided exploration.
Problem
Exploration in sequential (one-by-one) multi-agent policy updating for heterogeneous cooperative tasks is under-studied; traditional divergence measures like KL are unstable and can suppress exploration.
Approach
Adds intra-agent (between episodes) and inter-agent (between current and preceding agent) policy divergence as intrinsic rewards, measured with a conditional Cauchy-Schwarz divergence estimated via kernel density/Gram matrices, which provides a deterministic lower bound and implicitly maximizes 2nd-order Rényi entropy.
Key results
- MADPO consistently outperforms MAPPO, HAPPO, HATRPO and A2PO on 10 MA-Mujoco scenarios, with largest gains when the number of agents is large.
- Conditional CS divergence beats KL-divergence and policy entropy incentives in ablations, with narrower IQM confidence intervals.
- MADPO adds only negligible wall-clock overhead relative to HAPPO and other baselines (Table 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mutual policy divergence maximization | sequential MARL · MA-Mujoco | episodic reward | Helped |
| use mutual policy divergence maximization | sequential MARL · Bi-DexHands | episodic reward | Helped |
| replace policy entropy incentive with conditional CS policy divergence (ablation) | MADPO · MA-Mujoco and Bi-DexHands | episodic reward | Helped |
| replace KL divergence with conditional CS divergence (ablation) | MADPO · MA-Mujoco | episodic reward | Helped |
| replace KL divergence with conditional CS divergence (ablation) | MADPO · aggregate evaluation | stability | Helped |
| remove inter-agent policy divergence (ablation) | MADPO · parameter sensitivity experiments | episodic reward | Hurt |
| remove intra-agent policy divergence (ablation) | MADPO · parameter sensitivity experiments | episodic reward | Hurt |
| set λ to 0.5, increasing the influence of inter-agent divergence (ablation) | MADPO · parameter sensitivity experiments | episodic reward | Hurt |
| use random rather than default or specialization-based agent updating order (ablation) | MADPO · MA-Mujoco | episodic reward | Helped |
| use conditional CS divergence instead of no intrinsic reward (ablation) | MADPO · exploration incentive comparison | episodic reward | Helped |
| use KL divergence instead of no intrinsic reward (ablation) | MADPO · exploration incentive comparison | episodic reward | Helped |
Surprising findings
- A random agent updating order outperformed default and specialization-based orders in MA-Mujoco.
Limitations
- Memory cost of restoring previous-episode information grows as the number of agents increases.
- Method is somewhat sensitive to the hyperparameters sigma and lambda.
- Evaluated only on two multi-joint continuous-control benchmarks.
Experimental setup
Datasets: Multi-Agent Mujoco Bi-DexHands
Benchmarks: Multi-Agent Mujoco Bi-DexHands
Metrics: episodic reward IQM sample efficiency (rliable) wall time
Baselines: MAPPO HAPPO HATRPO A2PO KL-divergence incentive policy entropy incentive
Builds on: HAPPO MAPPO A2PO HATRPO conditional Cauchy-Schwarz divergence
Closest papers by meaning
- The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration (NeurIPS 2026 · similarity 0.88)
- Optimistic Multi-Agent Policy Gradient (ICML 2024 · similarity 0.88)
- Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.88)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.87)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.