Research map / Multi-agent systems and games / Cooperative multi-agent RL
ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConcaveQ, a value factorization MARL method using input-concave mixing networks with iterative action selection during training and soft actor-critic local policies for decentralized execution, evaluated on predator-prey and SMAC.
Problem
Monotonic value factorization (IGM-based, e.g. QMIX) has limited representation expressiveness and fails on non-monotonic cooperative tasks requiring coordination.
Approach
Replace the monotonic mixing network with an input-concave mixing network (non-negative hypernetwork weights, convex non-decreasing ReLU activations), find joint optimal actions during training via iterative per-agent action selection exploiting concavity, and train local soft actor-critic policies for decentralized execution.
Key results
- On Predator-Prey with harsher punishment (p=-0.5, -1.5, -2.0), only ConcaveQ and WQMIX keep high rewards within 60 episodes, with ConcaveQ converging faster than WQMIX.
- On SMAC hard and super-hard maps, ConcaveQ converges faster and reaches higher win rates, with a large margin on 6h_vs_8z.
- Ablations on 3s_vs_5z show removing iterative action selection, the soft policy network, or the concave mixer each lowers performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConcaveQ instead of monotonic-factorization baselines | ConcaveQ · Predator-Prey, p=0 | mean reward | No clear effect |
| Use ConcaveQ instead of monotonic-factorization baselines | ConcaveQ · Predator-Prey, p=-0.5, p=-1.5, and p=-2.0 | mean reward | Helped |
| Use ConcaveQ instead of the compared baselines | ConcaveQ · SMAC hard and super-hard maps | test win rate | Helped |
| Use ConcaveQ instead of the compared baselines | ConcaveQ · SMAC 6h_vs_8z | test win rate | Helped |
| Replace the concave mixing network with a 4-layer monotonic network (ablation) | ConcaveQ · SMAC 3s_vs_5z | reward | Hurt |
| Remove iterative action selection (ablation) | ConcaveQ · SMAC 3s_vs_5z | reward | Hurt |
| Remove the soft policy network (ablation) | ConcaveQ · SMAC 3s_vs_5z | reward | Hurt |
| Remove both iterative action selection and the soft policy network (ablation) | ConcaveQ · SMAC 3s_vs_5z | reward | Hurt |
| Remove the central Q* estimator (ablation) | ConcaveQ · SMAC 3s_vs_5z | reward | Hurt |
Surprising findings
- The paper claims monotonic factorization may recover only an arbitrarily small fraction of optimal joint actions, as the number of agents grows.
Limitations
- Concave mixing requires non-negative hypernetwork weights, which is somewhat restrictive per the authors.
- Iterative action selection scales as O(|U|*n) and the optimal joint action is not directly recoverable from local utilities, so execution relies on a learned soft policy.
- Experiments are limited to 3 seeds and a subset of SMAC maps and predator-prey.
Experimental setup
Datasets: SMAC Predator-Prey
Benchmarks: SMAC 3s_vs_5z SMAC 5m_vs_6m SMAC 6h_vs_8z SMAC 27m_vs_30m SMAC MMM2 SMAC corridor Predator-Prey
Metrics: test win rate average test reward
Baselines: QMIX WQMIX PAC QPLEX ResQ FOP
Builds on: QMIX WQMIX soft actor-critic VDN
Closest papers by meaning
- ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization (NeurIPS 2022 · similarity 0.91)
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (ICML 2018 · similarity 0.90)
- Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization (NeurIPS 2021 · similarity 0.90)
- Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning (AAAI 2026 · similarity 0.90)
- Optimistic Value Instructors for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.