Research map / Multi-agent systems and games / Cooperative multi-agent RL
Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAST, a dynamic sparse training framework for value-based deep MARL that uses hybrid TD(λ) targets, Soft Mellowmax operator, and a dual replay buffer to train ultra-sparse MARL agents from scratch with up to 20x FLOP reduction.
Problem
Training deep value-based MARL agents is computationally expensive, and naively applying dynamic sparse training fails because sparse networks cause unreliable bootstrapped value targets and unstable, policy-inconsistent sample distributions in multi-agent settings.
Approach
Combine RigL-style gradient-based topology evolution with three value-learning fixes for sparse models: a delayed hybrid TD(λ) target, the Soft Mellowmax operator to reduce overestimation without joint-action cost, and a dual FIFO replay buffer (large off-policy plus small on-policy) to reduce policy inconsistency.
Key results
- MAST-QMIX reaches about 98.9% of dense performance on average across four SMAC maps at roughly 92% sparsity (0.086x model size).
- FLOPs for training and inference are reduced by up to 20x (MAST-QMIX on 2s3z), averaging 10x, 9x and 8x for QMIX, WQMIX and RES.
- Under the same sparsity, static sparse, SET, RigL and RLx2 baselines reach much lower relative performance (e.g., RigL average 64.6% for QMIX).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAST instead of sparse training baselines | QMIX on SMAC · SMAC 3s5z | win rate (%): 45.2 -> 99.0 | Helped |
| Use MAST instead of sparse training baselines | WQMIX on SMAC · SMAC 3s5z | win rate (%): 70.7 -> 96.1 | Helped |
| Use MAST instead of sparse training baselines | RES on SMAC · SMAC 64* | win rate (%): 83.3 -> 104.9 | Helped |
| Use MAST with sparse models rather than dense models | QMIX, WQMIX, and RES · SMAC | performance degradation: under $3\%$ | No clear effect |
| Add the Soft Mellowmax operator to sparse QMIX | RigL-QMIX on SMAC 3s5z · SMAC 3s5z | win rate | Helped |
| Add dual replay buffers to sparse QMIX | RigL-QMIX · SMAC 3s5z | reward | Helped |
| Use dual replay buffers in dense training | QMIX · SMAC | training stability | No clear effect |
| Use hybrid TD($\lambda$) targets rather than one fixed multi-step length | sparse value-based MARL · SMAC | performance | Helped |
| Use a fixed random sparse topology instead of evolving a MAST topology | sparse QMIX · SMAC 3s5z | performance | Hurt |
| Use RigL's winning-ticket topology instead of MAST's | sparse QMIX · SMAC 3s5z | performance | Hurt |
| Use adaptive sparsity allocation rather than manually set agent sparsity patterns | MAST · SMAC 3s5z | performance | Helped |
| Use MAST sparse training instead of tiny dense networks | QMIX, WQMIX, and RES · SMAC average | win rate (%): 77.1 -> 98.9; 77.4 -> 98.1; 93.2 -> 100.6 | Helped |
Surprising findings
- RLx2, a single-agent sparse training method, performs poorly in MARL, averaging 29.8% relative performance for QMIX.
- RigL's winning ticket performs poorly, similar to a random ticket, while MAST's learned topology performs well.
Limitations
- Several hyperparameters (topology update, TD(lambda), dual buffer sizes) must be tuned for each setting.
- Unstructured sparsity does not directly translate to reduced running time.
- Evaluated mainly on SMAC easy maps with a few algorithms.
Experimental setup
Datasets: SMAC MAMuJoCo
Benchmarks: SMAC MAMuJoCo
Metrics: win rate FLOPs (train and test) model size
Baselines: Tiny dense network Static sparse (SS) SET RigL RLx2
Builds on: RigL QMIX Soft Mellowmax operator TD(lambda)
Closest papers by meaning
- Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (NeurIPS 2021 · similarity 0.91)
- SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning (AAAI 2025 · similarity 0.87)
- In value-based deep reinforcement learning, a pruned network is a good network (ICML 2024 · similarity 0.87)
- Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.