Research map / Multi-agent systems and games / Cooperative multi-agent RL
Major-Minor Mean Field Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Major-Minor Mean Field Control (M3FC), a discrete-time MFC model with one major agent plus many minor agents, and M3FMARL, which runs single-agent policy gradient (PPO) on finite multi-agent systems to approximate the M3FC MDP policy gradient.
Problem
Standard mean field control assumes only many homogeneous, weakly interacting minor agents, which is too restrictive for systems that also contain major agents or global environment states; cooperative MARL with many agents also suffers from combinatorial complexity and credit assignment issues.
Approach
Extend discrete-time MFC with a single major agent whose actions affect the minor population's mean field; the minor-agent policy is replaced by a mean-field action (a desired joint state-action distribution), so the problem reduces to a single-agent MDP whose complexity is independent of N. The proposed M3FMARL trains PPO on finite-agent systems directly and is shown to approximate the M3FC MDP p
Key results
- On N=20 agents, M3FPPO achieves the best return on 2G (-22.2), Beach (-303.5) and Foraging (1479.4) versus IPPO and MAPPO, while IPPO is best on Formation (-51.1).
- M3FPPO remains applicable with as few as 5 minor agents, and policies transferred to N=2..50 converge to the N=500 limit performance.
- In Beach, M3FPPO learns to accumulate up to 70% of agents on the bar.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M3FPPO instead of the strongest baseline, MAPPO | 2G · 2G | mean episode return: -26.0 -> -22.2 | Helped |
| Use M3FPPO instead of the strongest baseline, IPPO | Formation · Formation | mean episode return: -51.1 -> -63.9 | Hurt |
| Use M3FPPO instead of the strongest baseline, MAPPO | Beach · Beach | mean episode return: -342.9 -> -303.5 | Helped |
| Use M3FPPO instead of the strongest baseline, M3FA2C | Foraging · Foraging | mean episode return: 1398.0 -> 1479.4 | Helped |
| Use M3FPPO instead of the strongest baseline, MAPPO | Potential · Potential | mean episode return: -26.7 -> -31.3 | Hurt |
| Sample M3FC actions per agent (decentralized execution) instead of using one shared, correlated M3FC action (ablation) | M3FMARL · Beach and other evaluated settings | mean episode return | No clear effect |
Surprising findings
- IPPO achieves the best return on the Formation problem, outperforming M3FPPO there.
- Decentralized execution via per-agent action sampling has little to no effect and can marginally improve performance.
Limitations
- Theoretical O(1/sqrt N) convergence rate proven only for finite state spaces; non-finite case left open.
- Single major agent in the main model; multiple major agents only stated as straightforward extension.
- Continuous state/action spaces require binning the mean field into histograms, which limits scalability.
Experimental setup
Benchmarks: 2G Formation Beach Foraging Potential
Metrics: mean episode return
Baselines: IPPO MAPPO M3FA2C
Builds on: mean field control proximal policy optimization multi-agent PPO independent PPO
Closest papers by meaning
- Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior (ICLR 2024 · similarity 0.89)
- Learning Discrete-Time Major-Minor Mean Field Games (AAAI 2024 · similarity 0.88)
- Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics (NeurIPS 2025 · similarity 0.88)
- Learning Mean Field Control on Sparse Graphs (ICML 2025 · similarity 0.88)
- Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean Field Games (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.