Research map / Multi-agent systems and games / Cooperative multi-agent RL
SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SrSv, a Transformer-based MARL framework that combines sequential action rollout with per-agent value estimation computed from the decoder's attention matrix, and evaluates it on SMAC, MA-MuJoCo and DubinsCar with up to 1024 agents.
Problem
Cooperative multi-agent RL in large-scale systems suffers from credit assignment difficulty and poor scalability as agent populations vary; existing value decomposition, sequential update, and sequential rollout methods each lack either individual value estimation or sequential action modeling.
Approach
Combine the autoregressive sequential action rollout of MAT with individual-centric sequential value estimation: each agent's value is computed by attention-weighting shared-MLP outputs over the decoder sequence (using the last attention row), so value estimates reflect predecessor actions and successor policies.
Key results
- On the 3s5z SMAC task, SrSv reaches 50% win rate in 1.12M timesteps versus 1.30M for MAPPO, 1.62M for A2PO and 2.00M for MAT.
- On DubinsCar with 1024 agents (trained on 8 agents, 100 epochs), SrSv reaches 0.30 reach ratio versus 0.16 for MAT and MAPPO.
- SrSv reaches 100% win rate on 3s5z at 3.56M timesteps, versus 4.92M for MAT, and is the only method besides MAT to do so.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SrSv instead of MAT | DubinsCar models trained with 8 agents · DubinsCar, eval 512 agents | reach ratio: 0.25 -> 0.34 (100 epochs); 0.68 -> 0.81 (300 epochs) | Helped |
| use SrSv instead of MAPPO or A2PO | MARL training on SMAC 3s5z · SMAC 3s5z | timesteps to 50% win rate: 1.12M (59m) vs MAPPO 1.30M (1h), A2PO 1.62M (1h28m), and MAT 2.00M (1h42m) | Helped |
| use SrSv instead of MAT | MARL training on SMAC 3s5z · SMAC 3s5z | timesteps to 100% win rate: 3.56M vs 4.92M timesteps | Helped |
| replace self-attention with MLPs (ablation) | SrSv on SMAC 3s5z · SMAC 3s5z | win rate | Hurt |
| replace self-attention with a GNN (ablation) | SrSv on SMAC 3s5z · SMAC 3s5z | win rate | Hurt |
| use only the encoder and remove the autoregressive decoder (ablation) | SrSv on SMAC 3s5z · SMAC 3s5z | win rate | Hurt |
| use only the decoder and remove the encoder (ablation) | SrSv on SMAC 3s5z · SMAC 3s5z | win rate | Hurt |
| remove successor-policy information from sequential value estimation (ablation) | SrSv on cooperative multi-agent tasks · SMAC | training efficiency | Hurt |
| increase training population from 8 to 16 agents (ablation) | SrSv · DubinsCar, eval 1024 agents after 100 epochs | reach ratio: 0.30 | Mixed |
| increase training population from 8 to 32 agents (ablation) | SrSv · DubinsCar, eval 1024 agents after 100 epochs | reach ratio: 0.22 | Mixed |
Surprising findings
- Removing the policy-distribution term from the value estimate (SrSv w/o pi) falls between full SrSv and MAT in training efficiency rather than matching SrSv.
- Training on more agents (16 or 32) reduced SrSv's scalability at test time under the same 100 epochs of training.
Limitations
- Scalability at large agent counts is shown only on DubinsCar, a simple navigation task.
- Performance gains over MAT on SMAC win rate are slight; the main benefit is in dead ratio and training efficiency.
- Scaling to larger training populations reduced test-time scalability in the reported experiments.
Experimental setup
Datasets: SMAC Multi-Agent MuJoCo DubinsCar
Benchmarks: SMAC Multi-Agent MuJoCo DubinsCar
Metrics: win rate dead ratio average step reward reach ratio SRT ATT
Baselines: MAPPO A2PO MAT HAPPO QMIX
Builds on: Multi-Agent Transformer Heterogeneous-Agent PPO A2PO Generalized Advantage Estimation Proximal Policy Optimization
Closest papers by meaning
- Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training (NeurIPS 2024 · similarity 0.87)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.87)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.87)
- Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games (ICML 2025 · similarity 0.87)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.87)
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.