Research map / Multi-agent systems and games / Cooperative multi-agent RL
Maximum Entropy Heterogeneous-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HASAC, a maximum-entropy heterogeneous-agent actor-critic for cooperative MARL with monotonic improvement and QRE convergence guarantees, plus the MEHAML template generalizing it.
Problem
Cooperative MARL methods maximizing the standard joint reward converge prematurely to suboptimal Nash equilibria, and existing on-policy (HAPPO/HATRPO) and off-policy methods suffer from sample inefficiency or instability; prior MaxEnt MARL methods lack general convergence guarantees.
Approach
Embed cooperative MARL in a probabilistic graphical model to derive a maximum-entropy MARL objective, then perform heterogeneous-agent soft policy iteration (sequential per-agent KL updates) yielding a soft actor-critic algorithm (HASAC) with monotonic improvement and convergence to quantal response equilibrium; generalize to the MEHAML mirror-learning template.
Key results
- HASAC achieves the best performance in 31 out of 35 tasks across all benchmarks compared to SOTA MAPG methods.
- On SMAC, HASAC reaches 97.5% on 8m_vs_9m and 90.0% on 5m_vs_6m, and over 90% win rate on 7 of 8 evaluated maps.
- In a single-state matrix game, HASAC escapes the suboptimal (A,A) equilibrium that MAPPO and HAPPO converge to, reaching the Pareto-optimal (C,C) with appropriate temperature.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn stochastic policies with the MaxEnt objective instead of deterministic policies with fixed Gaussian exploration noise (ablation) | HASAC · Ant-v2 4x2 | reward | Helped |
| learn stochastic policies with the MaxEnt objective instead of deterministic policies with fixed Gaussian exploration noise (ablation) | HASAC · Ant-v2 4x2 | training stability | Helped |
| set temperature alpha to 1 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.0000 | Hurt |
| set temperature alpha to 2 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.0000 | Hurt |
| set temperature alpha to 5 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.0076 | No clear effect |
| set temperature alpha to 10 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.9555 | Helped |
| set temperature alpha to 15 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.7368 | Helped |
| set temperature alpha to 20 (ablation) | HASAC · Matrix game | probability of action C at convergence: 0.4697 | Mixed |
| increase temperature alpha (ablation) | HASAC · Matrix game, first policy update | probability of action C: 0.0009 -> 0.3009 | Helped |
| use stochastic policies and exploration encouraged by the MaxEnt objective (ablation) | HASAC · Matrix game | equilibrium reward | Helped |
| use HASAC instead of the strongest listed baseline | HASAC · SMAC 8m_vs_9m | win rate: 83.8(4.1) -> 97.5(1.2) | Helped |
| use HASAC rather than HAPPO | HASAC · SMAC 5m_vs_6m | win rate: 77.5(7.2) -> 90.0(3.9) | Helped |
Surprising findings
- In the matrix game, an overly large temperature alpha leads to poor performance because the entropy term dominates reward maximization.
- A deterministic variant with Gaussian exploration noise shows high variance across runs, whereas stochastic HASAC is more stable.
Limitations
- FOP-style IGO assumption limits prior methods; HASAC designed for continuous actions and requires Gumbel-Softmax for discrete tasks.
- Temperature alpha must be tuned (fixed or auto-tuned), and performance is sensitive to it.
Experimental setup
Datasets: Bi-DexHands Multi-Agent MuJoCo SMAC Google Research Football Multi-Agent Particle Environment Light Aircraft Game
Benchmarks: Bi-DexHands Multi-Agent MuJoCo SMAC Google Research Football Multi-Agent Particle Environment Light Aircraft Game
Metrics: win rate episode return
Baselines: HAPPO HATRPO MAPPO HATD3 QMIX FOP
Builds on: HAML HAPPO HATRPO SAC FOP
Closest papers by meaning
- FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (ICML 2021 · similarity 0.89)
- Divergence-Regularized Multi-Agent Actor-Critic (ICML 2022 · similarity 0.89)
- Towards Principled Unsupervised Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Sample-Efficient Multi-Agent RL: An Optimization Perspective (ICLR 2024 · similarity 0.88)
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.87)
- S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.