Research map / Multi-agent systems and games / Cooperative multi-agent RL

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning

ICML 2025 · Multi-agent systems and games › Cooperative multi-agent RL · approach: Cooperative MARL algorithms

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies target estimation error from extrapolation in large joint action spaces as a key bottleneck for off-policy cooperative MARL and proposes annealed multi-step bootstrapping, averaged Q-targets, and restricted action representation to mitigate it.

Problem

Off-policy cooperative MARL with joint Q-functions degrades as the number of agents grows, because exponentially large joint action spaces cause extrapolation errors in TD targets (target estimation error), which value factorization does not resolve.

Approach

Decomposes Q-error into target approximation, target estimation, and optimality terms, and shows target estimation error dominates in MARL; mitigates it with Peng's Q(lambda) targets whose lambda is annealed from 1 to a final value, ensemble-averaged Q-targets, and restricting joint-action influence (Sigmoid-bounded QPLEX weights, categorical action mapping for MADDPG).

Key results

• AEFACMAC and AEMADDPG-RAR significantly improve over FACMAC and MADDPG on four SMAC maps, with the largest gains on Super Hard maps.
• AEQMIX improves QMIX across 15 SMACv2 maps, especially on the hardest tasks such as 10 vs 11 and 20 vs 23 where QMIX's win rate is low.
• Ablations show larger ensemble size M consistently improves QMIX performance, and larger QMIX hidden sizes worsen performance, which the authors attribute to extrapolation error.

Measured effects reported in the paper

Surprising findings

• Increasing QMIX hidden size worsens performance, suggesting larger networks are more prone to extrapolation errors.
• Mapping joint actions to a continuous g(a) for MADDPG gives no improvement; only categorical restriction helps.

Limitations

• Restricting action representation limits expressiveness of the joint action space.
• Larger lambda leads to biased policies with fixed behavior policy, requiring annealing.
• Theoretical guarantees assume monotonic factorization (EPC), which excludes non-monotonic methods.

Experimental setup

Datasets: SMAC SMACv2 Google Research Football MPE spread

Benchmarks: SMAC SMACv2 Google Research Football MPE spread

Metrics: mean test win rate normalized return target estimation error

Baselines: QMIX FACMAC MADDPG QPLEX VDN Central-Q

Builds on: QMIX FACMAC MADDPG QPLEX Peng's Q(lambda)

Closest papers by meaning

• Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.90)
• Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.89)
• ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.89)
• Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.89)
• Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.88)
• Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.88)