Research map / Multi-agent systems and games / Cooperative multi-agent RL
Acting Beyond Learning: Imagination-Assisted Decision-Making in the Visual-based Multi-Agent Cooperative Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLWPO, a model-based MARL method that learns a contrastive latent world from pixel observations and optimizes policies with a hybrid of model-free MASAC and adaptive-rollout model-based self-imagination, evaluated on PettingZoo visual tasks.
Problem
Multi-agent cooperative control from visual observations suffers from low sample efficiency and difficulty in learning good state representations under non-stationarity and partial observability.
Approach
Learns decentralized per-agent representation modules plus a shared VAE-based state inference module, trains the latent world with a contrastive variational bound (InfoNCE) instead of observation reconstruction, and uses an adaptive per-agent rollout length based on teammate-model prediction errors, combined with model-free MASAC in a heuristic policy optimization scheme.
Key results
- On Cooperative Pong at 1K episodes, CLWPO reached 68.66 ± 10.73 return vs. 59.05–59.24 for MAPPO/MASAC and 58.15 for MAMBA (as reported in Table 1 averages).
- On Pistonball 5agents at 2K episodes, CLWPO needed 31.00 ± 4.52 episode steps vs. 83.61 (MAPPO), 49.33 (MASAC), 65.87 (MAMBA).
- On Pursuit with 6 pursuers and 20 evaders, all methods including CLWPO failed to capture all evaders within the 500-step limit, so improvement was not evident.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLWPO instead of MAMBA | Cooperative Pong · Cooperative Pong, 2K Episode Steps | average episode steps: 68.32 ± 12.96 vs 61.30 ± 7.15 | Helped |
| Use CLWPO instead of MASAC | Pistonball 5agents · Pistonball 5agents, 2K Episode Steps | average episode steps: 31.00 ± 4.52 vs 49.33 ± 20.98 | Helped |
| Use CLWPO instead of MASAC | Pistonball 6agents · Pistonball 6agents, 2K Episode Steps | average episode steps: 50.16 ± 18.46 vs 68.59 ± 13.93 | Helped |
| Use CLWPO instead of MAPPO | Cooperative Pong · Cooperative Pong, 1K Episode Steps | average episode steps: 68.66 ± 10.73 vs 59.05 ± 4.82 | Helped |
| Use CLWPO instead of MASAC | Pistonball 5agents · Pistonball 5agents, 1K Episode Steps | average episode steps: 42.47 ± 17.37 vs 54.16 ± 9.59 | Helped |
| Use CLWPO instead of MASAC | Pistonball 6agents · Pistonball 6agents, 1K Episode Steps | average episode steps: 55.93 ± 22.75 vs 68.59 ± 13.93 | Helped |
| Use CLWPO instead of other methods | Pursuit · Pursuit, 1K and 2K Episode Steps | average episode steps: 500 | No clear effect |
| Use an appropriate sequence length (ablation) | CLWPO · Pistonball 6agents | average return | Helped |
| Use an adaptive rollout length based on the initial rollout length k instead of a fixed rollout length (ablation) | CLWPO · Pistonball 6agents | average return | Helped |
| Add model-free MARL learning to model-based planning (ablation) | CLWPO policy optimization · Pistonball 6agents | average return | Helped |
| Set α_HPO to one so model-free learning and model-based planning impact training equally (ablation) | CLWPO policy optimization · Pistonball 6agents | average return | Helped |
| Replace contrastive learning with reconstruction of agents' observations (ablation) | CLWPO representation learning · Pistonball 6agents | average return | Hurt |
Surprising findings
- On the Pursuit task, all methods including CLWPO fail to finish within the 500-step limit, so the proposed method shows no clear advantage there.
Limitations
- Gains are not evident on the Pursuit task with sparse rewards and high difficulty.
- Evaluated only on PettingZoo visual tasks, not on broader cooperative, competitive, or realistic environments.
Experimental setup
Datasets: PettingZoo Butterfly (Cooperative Pong, Pistonball) PettingZoo SISL (Pursuit)
Benchmarks: PettingZoo Cooperative Pong PettingZoo Pistonball 5agents PettingZoo Pistonball 6agents PettingZoo Pursuit
Metrics: average episode return average episode steps
Baselines: MAMBA MASAC MAPPO CLWPO-rec_obs CLWPO-fixed_roll CLWPO-image
Builds on: MAMBA MASAC Dreamer PlaNet Zhang et al. 2021 model-based policy optimization bound
Closest papers by meaning
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.88)
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.88)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.87)
- Grounded Answers for Multi-agent Decision-making Problem through Generative World Model (NeurIPS 2024 · similarity 0.87)
- Open-World Reinforcement Learning over Long Short-Term Imagination (ICLR 2025 · similarity 0.87)
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.