Research map / Multi-agent systems and games / Cooperative multi-agent RL
Backpropagation Through Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Back-Propagation Through Agents (BPTA), which passes gradients through action dependencies in auto-regressive multi-agent policies, and instantiates it as BPPO on top of PPO.
Problem
Auto-regressive joint policies in cooperative MARL model forward action dependencies but ignore feedback from subsequent agents during policy updates, causing conflicting local policy-improvement directions and sub-optimal joint policies.
Approach
Unfold the agent execution sequence like BPTT and use the reparameterization trick (Gumbel-softmax for discrete actions) so gradients from subsequent agents' policies flow back through preceding agents' actions, yielding a conditional multi-agent policy gradient theorem with an explicit peer-learning term; BPPO applies this to a clipped PPO objective with reverse-order sequential updates.
Key results
- On the Climbing and Penalty matrix games, BPPO is the only method that converges to the Pareto-optimal equilibria in all games, while baselines converge to locally optimal policies.
- On MA-MuJoCo HalfCheetah-v2 6x1 and Walker2d-v2 6x1, BPPO outperforms baselines by a large margin where other algorithms fail to learn meaningful joint policies.
- On GRF 5-vs-5 full game, BPPO gains the highest score while baselines barely learn anything; results are averaged over five random seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BPTA feedback through action dependencies | Auto-regressive MAPG · Climbing and Penalty games | payoff | Helped |
| Project preceding actions to higher-dimensional vectors | ARMAPPO · Matrix games | payoff | Helped |
| Add BPTA feedback through action dependencies | Auto-regressive MAPG · SMACv2 | win rate | Helped |
| Add BPTA feedback through action dependencies | Auto-regressive MAPG · MA-MuJoCo | return | Mixed |
| Add BPTA feedback through action dependencies | Auto-regressive MAPG · Google Research Football academy scenarios | score | No clear effect |
| Add BPTA feedback through action dependencies | Auto-regressive MAPG · Google Research Football 5-vs-5 | return | Helped |
| Use simultaneous rather than sequential agent updates (ablation) | BPPO · Training schemes | performance | No clear effect |
| Project preceding-agent actions to higher-dimensional vectors (ablation) | ARMAPPO · Matrix games | payoff | Helped |
| Change the execution order to reverse or random (ablation) | BPPO · HalfCheetah-v2 6x1 | return | Mixed |
Surprising findings
- ARMAPPO, despite an auto-regressive policy, still fails to find the optimal joint policy in matrix games, but projecting preceding actions to higher-dimensional vectors (ARMAPPO w/ PROJ) converges to the optimum.
- Sequential and simultaneous update schemes for BPPO achieve similar performance in ablations.
Limitations
- Execution order is fixed to sequential in main experiments; the paper leaves adaptive ordering to future work.
- Performance gains on MA-MuJoCo are comparable to baselines in most tasks, with superior results only in certain scenarios.
- Gradient propagation relies on reparameterization (Gumbel-softmax for discrete actions), an approximation for discrete action spaces.
Experimental setup
Datasets: Climbing game Penalty game SMACv2 Multi-agent MuJoCo Google Research Football
Benchmarks: Climbing game Penalty game SMACv2 Multi-agent MuJoCo Google Research Football
Metrics: win rate episode return convergence to Pareto-optimal equilibrium
Baselines: MAPPO HAPPO ARMAPPO GCS
Builds on: PPO MAPPO HAPPO Back-Propagation Through Time GAE
Closest papers by meaning
- More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization (ICLR 2023 · similarity 0.89)
- ACE: Cooperative Multi-Agent Q-learning with Bidirectional Action-Dependency (AAAI 2023 · similarity 0.88)
- AgentMixer: Multi-Agent Correlated Policy Factorization (AAAI 2025 · similarity 0.87)
- Efficient Multi-agent Reinforcement Learning by Planning (ICLR 2024 · similarity 0.87)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.87)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.