Research map / Multi-agent systems and games / Cooperative multi-agent RL
STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAS, a spatial-temporal return decomposition method that uses temporal attention and Monte Carlo-approximated Shapley attention to redistribute a sparse episodic global reward into per-agent, per-step rewards for CTDE MARL with independent PPO.
Problem
Credit assignment in cooperative MARL when the global reward is revealed only at the end of an episode, so delayed rewards must be decomposed across both time steps and agents.
Approach
Combines temporal attention with causality masks (for Markovian per-step rewards) and a spatial Shapley attention that uses random masking to approximate marginal contributions, with Monte Carlo sampling of coalitions, to decompose the episodic return into per-agent per-step rewards that are then used to train independent PPO policies.
Key results
- On Alice & Bob, STAS reaches an average reward of about 200 and a treasure rate of 0.9 at 30k episodes, while SQDDPG reaches about 130 and 0.65 and COMA fails (treasure rate 0).
- In predator-prey (3 agents), STAS achieves a credit-distance Pearson coefficient of 0.3592 versus 0.2672 for SQDDPG, -0.1572 for QMIX and ~0 for COMA.
- Ablation indicates 3 Shapley samples already perform close to 5 samples, while 1 sample is worst and most unstable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STAS for delayed episodic rewards | cooperative agents in Alice & Bob · Alice & Bob | average reward: 200 | Helped |
| use STAS for delayed episodic rewards | cooperative agents in Alice & Bob · Alice & Bob | treasure rate: 0.9 | Helped |
| use 1 Shapley-value approximation sample instead of more samples (ablation) | STAS · Predator-prey (3 agents) | average reward | Hurt |
| increase Shapley-value approximation samples from 1 to 3 or 5 (ablation) | STAS · Predator-prey (3 agents) | average reward | Helped |
| use 3 Shapley samples rather than 5 (ablation) | STAS · Predator-prey (3 agents) | average reward | No clear effect |
| use 1 layer instead of multiple layers (ablation) | STAS-ML · Predator-prey (3 agents) | average reward | Hurt |
| increase the number of layers from 1 to 3 or 5 (ablation) | STAS-ML · Predator-prey (3 agents) | average reward | Helped |
| use 3 layers rather than 5 (ablation) | STAS-ML · Predator-prey (3 agents) | average reward | No clear effect |
Surprising findings
- STAS-ML outperforms STAS in predator-prey with three agents, suggesting that forgoing Shapley properties can help sample efficiency in small tasks.
- COMA fails entirely on Alice & Bob, achieving zero treasure rate.
Limitations
- Baseline set is limited (QMIX, COMA, SQDDPG), as the authors acknowledge.
- Results on a custom toy environment (Alice & Bob) and two MPE scenarios only.
- Evaluation of performance and variance is noisy for some baselines (QMIX, SQDDPG high variance).
Experimental setup
Datasets: Alice & Bob (custom environment) MPE cooperative navigation MPE predator-prey
Benchmarks: Alice & Bob MPE cooperative navigation MPE predator-prey
Metrics: average episode reward treasure reaching rate Pearson correlation between credit and inverse distance to p
Baselines: QMIX COMA SQDDPG STAS-ML
Builds on: QMIX SQDDPG COMA PPO Shapley value
Closest papers by meaning
- Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games (AAAI 2020 · similarity 0.88)
- Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning (AAAI 2022 · similarity 0.88)
- Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning (ICML 2024 · similarity 0.87)
- Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.86)
- Learning the Expected Core of Strictly Convex Stochastic Cooperative Games (NeurIPS 2024 · similarity 0.86)
- Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.