Research map / Multi-agent systems and games / Cooperative multi-agent RL
Investigating Relational State Abstraction in Collaborative MARL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARC, a centralized-critic actor-critic MARL method that abstracts observations into a spatial relational graph processed by an R-GCN shared across agents, improving sample efficiency and asymptotic performance on discrete grid and continuous tasks.
Problem
Sample inefficiency in collaborative multi-agent RL under spatially complex tasks where agents cannot communicate directly.
Approach
Transforms each agent's observation into a graph of agents and environment objects connected by translation-invariant relative spatial relations (left, right, top, bottom, adjacent, aligned), encodes it with a shared R-GCN and max-pooling, and feeds it to per-agent critic heads in a SAC-style actor-critic, so each critic sees only its own observation.
Key results
- On CPP and LBF-10x10-4a-4f-coop, MARC improves asymptotic performance by 69.9% and 35.2% over the best baselines.
- On LBF-15x15-8a-1f-coop, MARC reaches 99% performance after 5.9e5 steps, while MAAC needs 7.3 times as many steps.
- When trained on LBF-10x10-4a-4f-coop and tested with 3 agents, MARC retains 38% of performance versus 0% for MAPPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MARC's spatial relational critic instead of the baseline architecture | MARC versus baselines · LBF-15x15-8a-1f-coop | environment steps to reach 99% performance: 5.9e5 environment steps; MAAC takes 7.3 times the number of steps | Helped |
| use MARC's spatial relational critic instead of the strongest baseline | MARC · CPP | asymptotic performance: 69.9% | Helped |
| use MARC's spatial relational critic instead of the strongest baseline | MARC · LBF-10x10-4a-4f-coop | asymptotic performance: 35.2% | Helped |
| use MARC's spatial relational critic instead of MAPPO | MARC · LBF-10x10-4a-4f-coop | steps to reach 26% of maximum returns: 1e6 steps; MAPPO takes 5.6 times the number of steps | Helped |
| use MARC's relational observation encoder | MAPPO critic · CPP, 7x7 test version | sample efficiency and asymptotic performance | Helped |
| use only local relations instead of the default relation set (ablation) | MARC · discrete ablation tasks | relative performance: 10.6% of the performance achieved by the chosen architecture | Hurt |
| add local relations to the default relation set (ablation) | MARC · discrete ablation tasks | performance | No clear effect |
| represent all grid elements as entities instead of only agents and objects (ablation) | MARC · discrete ablation tasks | relative performance after 8e6 environment steps: 46.9% of the performance achieved by the chosen architecture | Hurt |
| replace R-GCN with R-GAT (ablation) | MARC · discrete ablation tasks | performance | No clear effect |
| replace R-GCN with GAT on a binary graph (ablation) | MARC · discrete ablation tasks | asymptotic performance: 23.4% of the chosen architecture's performance | Hurt |
| add finer octagonal-direction and adjacency relations (ablation) | MARC · Target continuous domain | asymptotic performance | Helped |
| increase the number of agents by 1 | MARC trained on LBF-10x10-4a-4f-coop · LBF generalization | performance: 93% vs. 88% for MAPPO | Helped |
Surprising findings
- Purely local relations reach only 10.6% of the default architecture's performance, and a GAT-based graph reaches only 23.4%.
- The R-GAT and R-GCN variants perform indistinguishably, suggesting learned edge importance is unnecessary.
Limitations
- Relies on the ability to extract entity positions and attributes from the observation, and performance is reported as sensitive to abstraction granularity in continuous domains.
- Continuous Target task shows MARC slightly below GA-AC and MAAC asymptotically, attributed to lost fine-grained proximity information.
- Results are from 3 random seeds with wide confidence intervals in some tasks.
Experimental setup
Datasets: Collaborative Pick and Place Level-based Foraging Wolfpack Target (multi-agent particle environment)
Benchmarks: Collaborative Pick and Place Level-based Foraging 10x10-4a-4f-coop Level-based Foraging 15x15-8a-1f-coop Wolfpack Target
Metrics: average return sample efficiency (environment steps to reach performance) generalisation performance across agent and fruit counts
Baselines: MAAC GA-AC InforMARL QMIX MAA2C MAPPO
Builds on: R-GCN SAC MAAC MAPPO EPyMARL
Closest papers by meaning
- Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.87)
- Bayesian Ego-graph inference for Networked Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems (AAAI 2025 · similarity 0.86)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.