Research map / Multi-agent systems and games / Cooperative multi-agent RL

Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning

NeurIPS 2024 · Multi-agent systems and games › Cooperative multi-agent RL · approach: Offline and networked MARL · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Scal-MAPPO-L, a scalable safe multi-agent constrained policy optimization method where each agent sequentially updates a κ-hop local policy with truncated advantages, removing reliance on the global state.

Problem

Safe multi-agent RL with centralized value functions relies on the global state and joint actions, which limits scalability and applicability in communication- or compute-constrained multi-agent systems with safety constraints.

Approach

Combines trust-region monotonic improvement bounds with bounds on advantage truncation under spatial correlation decay (Dobrushin-type) assumptions, yielding a per-agent local objective that depends only on κ-hop neighbors; agents update sequentially, and the authors prove joint reward improvement and safety constraint satisfaction, implemented as a MAPPO-Lagrangian variant.

Key results

• On Safe ManyAgent Ant tasks, Scal-MAPPO-L (using about half the agents' states via κ-hop truncation) shows nearly identical reward and cost curves to MAPPO-L (global state) and fewer constraint violations than IPPO and HAPPO.
• In the κ sensitivity study, κ=1 yields the lowest reward and highest cost, while κ>=3 substantially improves performance, approaching or in some cases outperforming MAPPO-L (e.g., κ=6 on Safe Ant 8x1).

Measured effects reported in the paper

Surprising findings

• Scal-MAPPO-L with truncated κ-hop observations sometimes outperforms the global-state MAPPO-L, which the authors attribute to difficulty extracting useful information from many messages.

Limitations

• Practical algorithm uses approximations (as in MAPPO-L) that may break the theoretical guarantees of the monotonic improvement and safety results.
• Assumptions 2.1-2.2 (spatial correlation decay) are hard to verify in practice; the authors use conservative empirically chosen values.
• Results are limited to Safe MAMuJoCo benchmark tasks.

Experimental setup

Datasets: Safe MAMuJoCo (Safe ManyAgent Ant 2x3, 3x2, 6x1; Safe Ant 8x1; Safe Coupled HalfCheetah 12x1; Safe Ant-v2 variants in ap

Benchmarks: Safe ManyAgent Ant Safe Ant Safe Coupled HalfCheetah Safe Ant-v2

Metrics: average episode reward average episode cost

Baselines: IPPO HAPPO MAPPO-L

Builds on: MAPPO-L HAPPO TRPO PPO Safe MAMuJoCo

Closest papers by meaning

• Multi-Agent First Order Constrained Optimization in Policy Space (NeurIPS 2023 · similarity 0.92)
• Scalable Safe Policy Improvement for Factored Multi-Agent MDPs (ICML 2024 · similarity 0.90)
• HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems (NeurIPS 2025 · similarity 0.89)
• Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.89)
• Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.89)
• Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.88)