Research map / Multi-agent systems and games / Cooperative multi-agent RL
Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Agent-Centric Actor-Critic (ACAC), a PPO-based CTDE method for asynchronous macro-action MARL that encodes each agent's history separately with timestep-aware encoders, aggregates them via attention for the critic, and uses a macro-level GAE.
Problem
Centralized training in asynchronous multi-agent RL with macro-actions is hard because agents make decisions at different times, and existing padding-based CTDE methods introduce misaligned, redundant, or spurious information in the centralized critic under sparse rewards.
Approach
Replace padding with per-agent history encoders (MLP+GRU over macro-observations with sinusoidal timestep embeddings) updated only when a new macro-observation arrives, aggregate them with attention into a centralized critic, and discount GAE by macro-decision steps rather than micro-timesteps.
Key results
- ACAC converges faster and reaches higher returns than padding-based macro-action baselines Mac-NIACC and Mac-IAICC across BoxPushing, Overcooked, and Overcooked-Rand (five seeds).
- On the harder Overcooked-Large environment (11x11 map, 6 agents), ACAC achieves optimal return in nearly all tasks while baselines often fail to learn.
- Ablation: duplicating padded macro-observations (ACAC-Duplicate) slows convergence or leads to suboptimal solutions; macro-level GAE discounting outperforms micro-level discounting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACAC instead of padding-based Mac-NIACC and Mac-IAICC | Asynchronous MARL · BoxPushing, Overcooked, and Overcooked-Rand | return | Helped |
| Use ACAC instead of padding-based baselines | Asynchronous MARL · Overcooked-Large | return | Helped |
| Use macro-actions instead of micro-actions | MARL · Overcooked | learning efficiency | Helped |
| Use the PPO-style loss instead of the baseline loss (ablation) | ACAC · BoxPushing, Overcooked, and Overcooked-Rand | training speed | Helped |
| Avoid duplicating the latest macro-observations in agent-centric history encoders (ablation) | ACAC · BoxPushing, Overcooked, and Overcooked-Rand | return | Helped |
| Use macro-level instead of micro-level λ-discounting in GAE (ablation) | ACAC · BoxPushing, Overcooked, and Overcooked-Rand | return | Helped |
Surprising findings
- Micro-action PPO baselines (ACACmicro, MAPPOmicro) perform relatively well on Overcooked, suggesting the underlying PPO algorithm matters.
Limitations
- Evaluated only on discrete macro-action environments; continuous action spaces left for future work.
- Lack of asynchronous MARL benchmarks combining continuous actions with macro-actions.
Experimental setup
Datasets: BoxPushing Overcooked Overcooked-Rand Overcooked-Large
Benchmarks: BoxPushing Overcooked Overcooked-Rand Overcooked-Large
Metrics: episode return convergence speed
Baselines: Mac-NIACC Mac-IAICC MAPPO (micro) ACAC-Vanilla ACAC-Duplicate
Builds on: MAPPO MacDec-POMDP GAE PopArt
Closest papers by meaning
- Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning (NeurIPS 2022 · similarity 0.94)
- Asynchronous Methods for Deep Reinforcement Learning (ICML 2016 · similarity 0.90)
- Actor-Attention-Critic for Multi-Agent Reinforcement Learning (ICML 2019 · similarity 0.89)
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms (ICML 2025 · similarity 0.88)
- Sequential Asynchronous Action Coordination in Multi-Agent Systems: A Stackelberg Decision Transformer Approach (ICML 2024 · similarity 0.88)
- Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.