Research map / Multi-agent systems and games / Cooperative multi-agent RL
GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GradPS, which identifies futile neurons in parameter-sharing MARL networks (neurons whose updates cancel from inter-agent gradient conflict) and clones them into groups of low-conflict agents, restoring them when clones become similar.
Problem
Parameter sharing in cooperative MARL gives sample efficiency but yields homogeneous policies; gradient conflicts among agents cancel neuron updates ("futile neurons"), hurting learning efficiency and expressiveness.
Approach
Measure neuron gradient conflict between agents, identify futile neurons with low neuron gradient efficiency, cluster agents by conflict (spectral clustering) into K groups, clone each futile neuron per group so each group shares an independent clone, and restore clones with minimal variance back to a normal shared neuron via a regularized SGD step.
Key results
- GradPS achieves the best test win rate versus FuPS, FuPS+ID, SePS, SNP, Kaleidoscope and NoPS on SMAC maps (e.g., 2c_vs_64zg, 27m_vs_30m, 3s5z_vs_3s6z, 5m_vs_6m, MMM2) and Predator-Prey small/medium/large.
- GradPS reduces the percentage of futile neurons relative to other PS methods and separates agents with hidden properties in Predator-Prey via gradient conflict clustering.
- GradPS improves QPLEX, DMIX, and RMIX as well as QMIX on 5m_vs_6m, MMM2, and Predator-Prey medium.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC 2c_vs_64zg | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC 27m_vs_30m | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC 3s5z_vs_3s6z | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC 5m_vs_6m | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC MMM2 | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · SMAC 1c3s8z_vs_1c3s9z | test win rate | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · Predator-Prey small | return | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · Predator-Prey medium | return | Helped |
| Use GradPS instead of parameter-sharing baselines | QMIX · Predator-Prey large | return | Helped |
| Add GradPS to QPLEX | QPLEX · SMAC 5m_vs_6m | test win rate | Helped |
| Add GradPS to DMIX | DMIX · SMAC MMM2 | test win rate | Helped |
| Add GradPS to RMIX | RMIX · Predator-Prey medium | return | Helped |
Surprising findings
- Gradient conflict is also described as potentially beneficial for policy diversity, since it signals that agents learned different skills.
Limitations
- Futile threshold alpha is a fixed constant (0.2) in experiments; adaptive thresholds are only suggested.
- Cloning increases the number of neurons, with cost only asserted to be low.
- Number of groups K is fixed as a hyperparameter; whether aggregate K reflects heterogeneity is left to future work.
Experimental setup
Datasets: SMAC Predator-Prey
Benchmarks: SMAC Predator-Prey
Metrics: test win rate return futile neuron percentage neuron gradient conflict
Baselines: FuPS FuPS+ID SePS SNP Kaleidoscope NoPS
Builds on: QMIX SePS SNP Kaleidoscope PCGrad
Closest papers by meaning
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization (NeurIPS 2024 · similarity 0.86)
- Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.86)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.