Research map / Multi-agent systems and games / Cooperative multi-agent RL
Towards Principled Unsupervised Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates multi-agent state entropy maximization via convex Markov games with joint, disjoint, and mixture objectives, proves theoretical relations, and proposes TRPE, a decentralized single-trial trust-region policy search that favors mixture entropy for unsupervised pre-training.
Problem
Unsupervised pre-training via state entropy maximization is well understood in single-agent RL but lacks a principled multi-agent formulation: which distributions to take entropy over, how much information agents need, and how to account for finite trials.
Approach
Casts multi-agent state entropy maximization as a convex Markov game with three objectives (joint, disjoint, mixture entropy), shows mixture entropy is a log(N)-approximate lower bound to joint entropy with finite-trial mismatch independent of the number of agents, and optimizes single-trial objectives with a decentralized importance-sampling trust-region method (TRPE).
Key results
- Mixture-entropy objective optimized via TRPE yields both high mixture and joint entropy in the secret room task at T=50, while joint and disjoint objectives fail to optimize mixture entropy.
- Pre-training with TRPE mixture objective leads to faster learning and higher performance than uniform initialization in sparse-reward secret room with T=50, while disjoint pre-training is harmful.
- In multi-agent Reacher, mixture and joint pre-training both give zero-shot performance after 2e4 samples, with mixture showing higher stability than joint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize the mixture entropy objective instead of joint or disjoint entropy | TRPE pre-training · Secret room, T=50 | joint entropy | Helped |
| optimize the mixture entropy objective instead of joint or disjoint entropy | TRPE pre-training · Secret room, T=50 | mixture entropy | Helped |
| pre-train with the mixture objective instead of uniform initialization | MA-TRPO · Secret room, T=150, worst-case goal | learning speed | Helped |
| pre-train with the joint objective instead of standard initialization | MA-TRPO · Secret room, T=150, worst-case goal | downstream performance | No clear effect |
| pre-train with the mixture objective instead of uniform initialization | MA-TRPO · Secret room, T=50, worst-case goal | downstream performance | Helped |
| pre-train with the disjoint objective instead of uniform initialization | MA-TRPO · Secret room, T=50, worst-case goal | downstream performance | Hurt |
| pre-train with the mixture objective instead of random initialization | MA-TRPO · Reacher, T=100, worst-case positions | zero-shot performance | Helped |
| pre-train with the joint objective instead of random initialization | MA-TRPO · Reacher, T=100, worst-case positions | zero-shot performance | Mixed |
| pre-train with the disjoint objective instead of random initialization | MA-TRPO · Reacher, T=100, worst-case positions | zero-shot performance | Hurt |
Surprising findings
- Disjoint-objective pre-training is harmful for downstream learning at short horizons, producing almost deterministic policies that overfit to states reachable without coordination.
- Joint-objective pre-training shows extremely high variance in zero-shot performance, sometimes no better than random initialization.
Limitations
- Monte-Carlo estimation of single-trial objectives may be sample-inefficient in high-dimensional tasks.
- Plug-in entropy estimator applies only to discrete spaces; continuous-domain entropy estimation is not addressed.
- Experiments are on small domains (secret room, 2-agent Reacher with discretized states).
Experimental setup
Datasets: secret room environment multi-agent Reacher
Benchmarks: secret room multi-agent Reacher
Metrics: state entropy (joint, mixture) downstream return sample complexity
Baselines: joint entropy objective disjoint entropy objective uniform initialization zero-mean initialization
Builds on: TRPO convex MDPs potential Markov games
Closest papers by meaning
- A Mixture Of Surprises for Unsupervised Reinforcement Learning (NeurIPS 2022 · similarity 0.91)
- SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation (ICLR 2025 · similarity 0.89)
- On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story (ICML 2025 · similarity 0.88)
- Constrained Ensemble Exploration for Unsupervised Skill Discovery (ICML 2024 · similarity 0.88)
- Maximum Entropy Heterogeneous-Agent Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.