Research map / Multi-agent systems and games / Cooperative multi-agent RL
Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiCo, which controls behavioral diversity (SND) in MARL to an exact target value by architecturally rescaling per-agent policy deviations from a parameter-shared policy, with no change to the learning objective.
Problem
Existing MARL diversity methods only boost diversity through intrinsic rewards or auxiliary losses, which change the learning objective and cannot set diversity to a specified value.
Approach
Represent each policy as a shared homogeneous component plus a per-agent deviation, and rescale the deviations by desired diversity divided by measured deviation diversity (SND), so the policy diversity equals the target exactly without any extra loss or reward.
Key results
- In Multi-Agent Navigation, constrained agents track the desired SND and reward increases with diversity input, with SND=0 agents settling at the midpoint between goals.
- In Dispersion, DiCo with SND_des=6 converges faster to the optimal food-assignment policy than unconstrained heterogeneous agents, which remain suboptimal.
- In Tag, DiCo at SND_des=0.6 almost doubles the obtained reward over the unconstrained heterogeneous baseline, with emergent man-marking and pinching strategies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Constrain diversity to a higher level | IPPO policies in Multi-Agent Navigation · Multi-Agent Navigation | mean reward | Helped |
| Constrain diversity to SND_des = 6 | MADDPG policies · Dispersion | training reward | Helped |
| Share policy parameters (SND_des = 0) | MADDPG policies · Dispersion | training reward | Hurt |
| Constrain diversity to SND_des = 5 | IDDPG policies · Sampling | training reward | Helped |
| Share policy parameters (SND_des = 0) | IDDPG policies · Sampling | training reward | Hurt |
| Constrain diversity to SND_des = 0.6 instead of SND_des = 0.3 | IPPO red-agent policies · Tag | training reward | Helped |
| Constrain diversity to SND_des = 0.3 | IPPO red-agent policies · Tag | training reward | No clear effect |
| Share policy parameters (SND_des = 0) | IPPO red-agent policies · Tag | training reward | No clear effect |
| Use a lower soft-update coefficient tau (ablation) | DiCo · Multi-Agent Navigation | variance of measured SND | Helped |
| Use a lower soft-update coefficient tau (ablation) | DiCo · Multi-Agent Navigation | mean SND tracking | Hurt |
| Use a higher soft-update coefficient tau (ablation) | DiCo · Multi-Agent Navigation | mean SND tracking | Helped |
| Use a higher soft-update coefficient tau (ablation) | DiCo · Multi-Agent Navigation | variance and stability of measured SND | Hurt |
Surprising findings
- Unconstrained heterogeneous agents in Tag perform only slightly better than fully homogeneous agents, and a DiCo model at the same diversity level as unconstrained (SND=0.3) matched that reward, with the gain only appearing at higher diversity.
Limitations
- Requires domain knowledge to choose the desired SND value; closed-loop PID search is proposed to mitigate this but is not the main evaluation.
- Training cost scales quadratically in the number of agents due to pairwise Wasserstein computations.
- SND is only defined for closed-form Wasserstein between Gaussian or deterministic policies, limiting the choice of diversity metric.
Experimental setup
Datasets: VMAS Multi-Agent Navigation VMAS Dispersion VMAS Sampling VMAS Tag Reverse Transport
Benchmarks: VMAS Multi-Agent Navigation VMAS Dispersion VMAS Sampling VMAS Tag
Metrics: mean instantaneous training reward SND diversity
Baselines: Homogeneous (parameter-shared) agents Unconstrained heterogeneous agents MADDPG IDDPG
Builds on: System Neural Diversity IPPO MADDPG BenchMARL
Closest papers by meaning
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.91)
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.89)
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization (AAAI 2024 · similarity 0.89)
- Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.88)
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.