Research map / Multi-agent systems and games / Cooperative multi-agent RL
Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes decentralized partially observable mean field control (Dec-POMFC), reduces it to a single-agent MDP over mean fields, and trains via centralized-training PPO with kernel (RBF) policy parametrizations, evaluated on Aggregation, Vicsek and Kuramoto swarm tasks.
Problem
Scalable cooperative multi-agent RL for decentralized, partially observable systems with many permutation-invariant agents, where existing MFC methods assume full observability and histogram discretization fails in high dimensions.
Approach
Reformulate finite decentralized partially observable MFC-type Dec-POMDPs via infinite-agent limit and open-loop control into a Dec-MFC MDP whose actions are lower-level policies parametrized by RBF kernels; train centrally with PPO (observing the mean field) and execute decentralized, with policy gradient approximation guarantees.
Key results
- Dec-POMFPPO is on par with IPPO and MAPPO on Aggregation, Vicsek and Kuramoto with N=200 agents.
- Vicsek alignment reaches polar order parameter above 0.9 across topologies, with the torus reaching about 0.99.
- RBF kernel parametrization outperforms histogram discretization for state dimensions higher than 3 in Aggregation; discretization stops learning at d=5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RBF kernel parametrization instead of discretization (ablation) | MFC for Aggregation · Aggregation d=4 | training performance | Helped |
| Use RBF kernel parametrization instead of discretization (ablation) | MFC for Aggregation · Aggregation d=5 | training performance | Helped |
| Use RBF kernel parametrization instead of discretization (ablation) | MFC for Aggregation · Aggregation d=2 and d=3 | training performance | Hurt |
| Use RBF kernel parametrization instead of discretization (ablation) | MFC for Aggregation · Aggregation d=5, 50 agents | wall-clock time per training step: 8.31 vs 76.33 | Helped |
| Train PPO without RL observations (ablation) | PPO for Vicsek · Vicsek torus | useful behavior | Hurt |
| Use the time-0 lower-level policy at every time step (ablation) | Dec-POMFPPO · Vicsek torus, N=200 | alignment | No clear effect |
| Transfer the learned policy to different initial conditions (ablation) | Dec-POMFPPO trained on Gaussian initialization · Vicsek torus, N=200 | alignment | Mixed |
| Use Dec-POMFPPO instead of IPPO or MAPPO | MARL · Aggregation, Vicsek, and Kuramoto | episode return | No clear effect |
Surprising findings
- PPO trained without mean field observations fails to learn useful behavior, highlighting the importance of mean field guidance during training.
- Kuramoto training is unstable and yields lower-quality behavior than Vicsek.
Limitations
- Theory limited to non-stochastic (deterministic) mean fields; common noise not analyzed.
- Kuramoto training unstable with lower-grade qualitative behavior.
- Sample efficiency not analyzed.
Experimental setup
Benchmarks: Aggregation Vicsek Kuramoto
Metrics: episode return polar order parameter
Baselines: IPPO MAPPO histogram/discretization-based MFC
Builds on: PPO IPPO MAPPO MFC MDP
Closest papers by meaning
- Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Major-Minor Mean Field Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.89)
- On the Convergence of Model Free Learning in Mean Field Games (AAAI 2020 · similarity 0.89)
- Decentralized Mean Field Games (AAAI 2022 · similarity 0.89)
- Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation (ICLR 2025 · similarity 0.87)
- Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.