Research map / Multi-agent systems and games / Cooperative multi-agent RL
Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUBSAMPLE-MFQ, which runs mean-field or standard Q-learning on a random k-agent subsystem and executes a decentralized randomized subsampled policy, with optimality gap O~(1/sqrt(k)) independent of n.
Problem
Cooperative MARL with a global agent and n homogeneous local agents has joint state-action spaces exponential in n, making exact Q-learning and mean-field Q-learning computationally prohibitive for large n.
Approach
Learn a Q-function on a randomly subsampled surrogate system of k local agents (via standard or mean-field value iteration with empirical Bellman operator), then deploy a decentralized randomized policy in which the global agent and each local agent sample k agents at each step; the TV-distance-based analysis gives an optimality gap decaying as 1/sqrt(k) independent of n.
Key results
- Theoretical bound on optimality gap of order O~(1/sqrt(k)), independent of the number of agents n, with high probability (1-1/(100e^k)).
- Simulations with n=8 show monotonic reward improvement as k approaches n, with reward for k<n only marginally below value iteration.
- Gaussian squeeze experiments show exponential computation-time improvement over mean-field value iteration and monotonic rewards for n=50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the subsampling parameter k | SUBSAMPLE-MFQ · Cooperative MARL (theoretical guarantee) | optimality gap: \tilde{O}(1/\sqrt{k}) | Helped |
| increase the subsampling parameter k (ablation) | SUBSAMPLE-MFQ · Constrained Exploration (n=8) | discounted cumulative reward | Helped |
| increase the subsampling parameter k (ablation) | SUBSAMPLE-MFQ · Gaussian Squeeze (n=50) | cumulative reward | Helped |
| use SUBSAMPLE-MFQ instead of mean-field value iteration | Gaussian Squeeze (n=8) · Gaussian Squeeze | computation time | Helped |
| use SUBSAMPLE-MFQ instead of mean-field value iteration | Gaussian Squeeze (n=50) · Gaussian Squeeze | cumulative reward | Helped |
| set k=n (ablation) | SUBSAMPLE-MFQ · Cooperative MARL | optimality gap | Helped |
Limitations
- Assumes a specific cooperative dynamic model with homogeneous local agents and tabular finite spaces; non-tabular results rely on linear MDP assumptions.
- Polynomial dependence on 1/(1-gamma) from using simple sampling rather than variance reduction.
- Experiments are small-scale (n up to 50) on a 2-core CPU server.
Experimental setup
Datasets: simulated constrained exploration grid environment simulated Gaussian squeeze environment
Benchmarks: constrained exploration Gaussian squeeze
Metrics: discounted cumulative reward optimality gap computation time
Baselines: mean-field value iteration standard value iteration (k=n)
Builds on: mean-field MARL Q-learning performance difference lemma Gamarnik correlation decay
Closest papers by meaning
- Major-Minor Mean Field Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.91)
- Decentralized Mean Field Games (AAAI 2022 · similarity 0.90)
- Mean Field Multi-Agent Reinforcement Learning (ICML 2018 · similarity 0.90)
- Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior (ICLR 2024 · similarity 0.89)
- Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.