Research map / Multi-agent systems and games / Cooperative multi-agent RL
Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CONS, a knowledge-sharing framework where MARL agents share both positive (best-action) and negative (worst-action) policy knowledge and softly update action probabilities with time-varying weights, plus targeted exploration, integrated into Q-learning without extra training cost.
Problem
Decentralized MARL agents need many interactions to learn coordinated policies; existing action advising methods share only positive advice, which can be suboptimal or adverse and hinder exploration in early learning.
Approach
Agents share both positive and negative knowledge (best and worst actions with probabilities), cautiously assimilate it by softly adjusting their action probabilities with weights that shift from negative to positive knowledge over training, and then explore in a targeted manner based on policy confidence. The method only changes action selection, not the underlying training.
Key results
- On PGM-6ag, CONS reaches the performance of AdHocTD and IQL in about half the episodes and eventually surpasses them; CONS uses about 30.1% and 33.5% of AdHocTD's request budget in PGM-6ag and PGM-3ag.
- On Find the Treasure, I2Q, SEQL and GA-Comm fail, while CONS learns faster and achieves higher reward than IQL and AdHocTD.
- On Cleanup, CONS uses only 8% of AdHocTD's requesting budget and eventually outperforms I2Q.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CONS knowledge sharing instead of the compared baselines | DRQN agents · PGM-6ag | team episode return and learning speed | Helped |
| Use CONS knowledge sharing instead of the compared baselines | DRQN agents · PGM-3ag | team episode return | Helped |
| Use CONS knowledge sharing instead of IQL and AdHocTD | DRQN agents · Find the Treasure | team episode return and learning speed | Helped |
| Use CONS knowledge sharing instead of I2Q | DRQN agents · Cleanup | team episode return | Helped |
| Use CONS instead of AdHocTD | DRQN agents · PGM-6ag | requesting budget utilization: 30.1% of the budget used by AdHocTD agents | Helped |
| Use CONS instead of AdHocTD | DRQN agents · PGM-3ag | requesting budget utilization: 33.5% of the budget used by AdHocTD agents | Helped |
| Use CONS instead of AdHocTD | DRQN agents · Cleanup | requesting budget utilization: 8% of the requesting budget used by AdHocTD agents | Helped |
| Add negative knowledge sharing (rather than omit it) (ablation) | CONS · PGM-6ag | team training reward | Helped |
| Add positive knowledge sharing (rather than omit it) (ablation) | CONS · PGM-6ag | team training reward | Helped |
| Remove targeted exploration and sample directly from the modified probability distribution (ablation) | CONS · PGM-6ag | team training reward | Hurt |
| Use cautious absorption of advice rather than strictly following it (ablation) | CONS · PGM-6ag | robustness to suboptimal advice | Helped |
| Remove targeted exploration (ablation) | CONS · PGM-6ag | team training reward in early learning | No clear effect |
Surprising findings
- Ablation suggests negative knowledge sharing may be more crucial than positive knowledge in challenging tasks.
- Strictly following advice can be better when the optimal strategy is evident or no suboptimal solutions exist, since targeted exploration may slow learning.
Limitations
- Relies on observation counters and value-based underlying algorithms, so limited to discrete tasks.
- May slightly slow learning when optimal strategy is evident or no suboptimal solutions exist.
- Evaluated only on small grid-world environments with few agents (3-6, plus a 20-agent scalability test).
Experimental setup
Datasets: Patient Gold Miner (modified ma-gym) Find the Treasure (modified ma-gym) Cleanup (Sequential Social Dilemma)
Benchmarks: Patient Gold Miner Find the Treasure Cleanup
Metrics: mean evaluating episode return budget utilization ratio of opened boxes and collected items
Baselines: I2Q SEQL GA-Comm AdHocTD IQL MAVEN
Builds on: AdHocTD DRQN independent Q-learning PSAF
Closest papers by meaning
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.89)
- Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.88)
- Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations (AAAI 2024 · similarity 0.88)
- Training Generalizable Collaborative Agents via Strategic Risk Aversion (NeurIPS 2026 · similarity 0.88)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.