Research map / Multi-agent systems and games / Cooperative multi-agent RL
Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMPE^2, a MARL method that learns variational state-belief representations from local observations with learnable agent-modelling filters, feeds them into the policy, and adds count-based adversarial intrinsic rewards on hashed beliefs, evaluated on MPE, LBF and RWARE.
Problem
Cooperative MARL under decentralized partial observability without communication, where agents must infer useful beliefs about the unobserved state and explore effectively in sparse-reward tasks.
Approach
Agents learn variational belief representations from their own observations, reconstructing other agents' observations filtered by learnable AM filters that remove redundant state information, with beliefs fed into the policy and trained w.r.t. value via a second critic; intrinsic rewards from SimHash counts over beliefs encourage agents to find novel observations that are also hard-to-predict tar
Key results
- SMPE^2 fully solves sparse-reward LBF tasks 2s-9x9-3p-2f and 4s-11x11-3p-2f where other methods fail, per the paper's claims.
- On RWARE tiny-2ag-hard, tiny-4ag-hard and small-4ag-hard, SMPE^2 outperforms SOTA methods including EOI, EMC and MASER.
- Ablation: removing the KL term yields beliefs that are easily separable by agent ID (99.3% logistic regression accuracy) versus 57.5% for full SMPE^2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove KL regularization (ablation) | SMPE² · LBF task (45th time step) | agent-ID classification accuracy on t-SNE embeddings: 57.5% -> 99.3% | Hurt |
| Remove AM filters wᵢ (ablation) | SMPE² · LBF task (45th time step) | agent-ID classification accuracy on t-SNE embeddings: 57.5% -> 80.8% | Hurt |
| Remove state modelling (ablation) | SMPE² · LBF tasks | episodic reward | Hurt |
| Remove intrinsic exploration (ablation) | SMPE² · LBF tasks | episodic reward | Hurt |
| Remove AM filters wᵢ (ablation) | SMPE² · LBF tasks | episodic reward | Hurt |
| Replace adversarial exploration with the standard exploration method (ablation) | SMPE² · LBF tasks | episodic reward | Hurt |
| Use only the wᵢ-conditioned critic for actor advantages (ablation) | SMPE² · Ablation tasks | episodic reward | Hurt |
| Lower λ_rec (ablation) | SMPE² · MPE Spread | episodic reward | Hurt |
| Increase λ_rec (ablation) | SMPE² · MPE Spread | episodic reward | Hurt |
| Set λ_norm to 0 (ablation) | SMPE² · Ablation tasks | episodic reward | Hurt |
| Vary λ_norm within [0.1, 1] (ablation) | SMPE² · Ablation tasks | performance | No clear effect |
| Learn state representations with respect to policy optimization (ablation) | SMPE² · Ablation tasks | episodic reward | Helped |
Surprising findings
- EMC and MASER fail to improve over QMIX on LBF, attributed to misleading intrinsic rewards built on low-value random-policy data.
- Without the KL term, agent beliefs are easily separable, indicating less cohesive beliefs and poorer joint exploration.
Limitations
- Evaluated only on MPE, LBF and RWARE; future work proposes stochastic settings and scalability as open issues.
- Relies on hyperparameters such as lambda, lambda_norm and beta for balancing losses and intrinsic reward.
Experimental setup
Datasets: MPE Spread MPE Double Speaker-Listener LBF RWARE
Benchmarks: MPE Spread LBF RWARE
Metrics: average episodic reward t-SNE logistic regression accuracy
Baselines: MAA2C COMA MAPPO ATM EOI EMC MASER SIDE MLIAM
Builds on: MAA2C LIAM SIDE EOI MAPPO
Closest papers by meaning
- Cooperative Exploration for Multi-Agent Deep Reinforcement Learning (ICML 2021 · similarity 0.90)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.90)
- AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.88)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.88)
- Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.