Research map / Multi-agent systems and games / Cooperative multi-agent RL
Optimistic Value Instructors for Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Optimistic Value Instructors (OVI), which add per-agent optimistic instructive value functions and instructive constraints to value-decomposition MARL (QMIX, QPLEX) to overcome relative overgeneralization.
Problem
Relative overgeneralization in cooperative multi-agent RL: decentralized agents and value-decomposition methods fail to identify and select the optimal joint action, often converging to sub-optimal equilibria due to representational limits or inaccurate sampling of optimal joint actions.
Approach
Learn per-agent optimistic instructive value functions (max-based over other agents, updated only on increases, with a complement factor beta on negative TD errors) that suggest optimal local actions; use these suggestions to add global and local instructive constraints that rectify the factored global value function and imitate the instructors.
Key results
- In the matrix game, QMIX-OVI and QPLEX-OVI learn the optimal joint action (a1,a1), with learned value 8.12 for QMIX-OVI, while baselines such as QMIX get stuck at a return of 6.
- On SMAC, QMIX-OVI significantly outperforms CW-QMIX and OW-QMIX and reaches high test win rates on all six maps.
- QPLEX-OVI improves convergence over QPLEX on 3s_vs_4z, 3s_vs_5z and 10m_vs_11m, but performs similarly on other maps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OVI to QMIX | QMIX · Matrix game | test return | Helped |
| Add OVI to QPLEX | QPLEX · Predator and prey | test return | Helped |
| Add OVI to QMIX | QMIX · SMAC | test win rate | Helped |
| Add OVI to QPLEX | QPLEX · SMAC (3s_vs_4z, 3s_vs_5z, 10m_vs_11m) | test win rate | Helped |
| Add OVI to QPLEX | QPLEX · SMAC (other evaluated maps) | test win rate | No clear effect |
| Use a small complement factor β rather than a value close to 1 (ablation) | QMIX-OVI and QPLEX-OVI · Evaluated tasks | performance | Helped |
Surprising findings
- QPLEX-OVI shows similar performance to QPLEX on several SMAC maps, which the authors attribute to QPLEX's complete IGM expressiveness.
Limitations
- Separately learning the joint action value function may cause sample inefficiency when the joint action space is extremely large.
- Only value-based methods are studied; extension to policy gradient is left for future work.
Experimental setup
Datasets: matrix game predator and prey StarCraft Multi-Agent Challenge maps (3s5z, 3s_vs_4z, 3s_vs_5z, 5m_vs_6m, 8m_vs_9m, 10m_vs_11m)
Benchmarks: matrix game predator and prey SMAC 3s5z SMAC 3s_vs_4z SMAC 3s_vs_5z SMAC 5m_vs_6m SMAC 8m_vs_9m SMAC 10m_vs_11m
Metrics: test return test win rate
Baselines: CW-QMIX OW-QMIX QPLEX QTRAN MAVEN QMIX
Builds on: QMIX QPLEX Distributed Q-learning Centrally-Weighted QMIX
Closest papers by meaning
- Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning (AAAI 2022 · similarity 0.90)
- Distributionally Robust Cooperative Multi-agent Reinforcement Learning with Value Factorization (ICLR 2026 · similarity 0.90)
- Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Optimistic Multi-Agent Policy Gradient (ICML 2024 · similarity 0.88)
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.