Research map / Multi-agent systems and games / Cooperative multi-agent RL
Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MANGER, which uses an RND novelty signal to assign extra per-agent Q-network updates on top of QMIX, with a separated critic layer for agent diversity, evaluated on SMAC and Google Research Football.
Problem
Deep MARL in cooperative tasks suffers from poor sample utilization (one update per interaction) and insufficient diversity among agent policies.
Approach
Use RND prediction error as an observation-novelty measure to decide how many extra updates each agent receives from a batch, and split the critic into shared and per-agent independent layers so extra updates affect only that agent's parameters, which also yields diverse behaviors.
Key results
- MANGER improves QMIX win rates on SMAC and GRF, outperforming QMIX, QPLEX and Qatten in most of the reported tasks (curves averaged over 5 runs).
- Mean number of extra updates per agent is reported to be below 0.5, so training time is not significantly increased relative to QMIX (e.g., 6h_vs_8z).
- Q-value cosine similarity among agents is lower under MANGER than under QMIX, indicating more differentiated agent behaviors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply novelty-guided sample reuse with RND-based per-agent updates | QMIX (MANGER) · SMAC tasks | win rate | Helped |
| Apply novelty-guided sample reuse with RND-based per-agent updates | QMIX (MANGER) · GRF tasks | win rate | Helped |
| Apply novelty-guided sample reuse with RND-based per-agent updates | QMIX (MANGER) · SMAC-V2 tasks | performance | Helped |
| Apply novelty-guided sample reuse with RND-based per-agent updates | MANGER · 6h_vs_8z | training time | No clear effect |
| Update agents at different frequencies using novelty | MANGER · SMAC tasks | Q-value similarity between agents | Helped |
| Replace novelty-guided updates with fixed multiple updates (ablation) | QMIX with separate Q values · 6h_vs_8z | win rate | Hurt |
| Add a separate Q-value module (ablation) | QMIX · 6h_vs_8z | win rate | Hurt |
| Use RND to target extra updates to novel samples rather than only increasing average update counts (ablation) | QMIX with separate Q values · 6h_vs_8z | win rate | Helped |
Surprising findings
- Adding a separate Q module to QMIX alone does not match MANGER, which the authors take as evidence that gains come from the RND-targeted repeated updates rather than the architecture change.
Limitations
- Novelty is used as a proxy for update need; no theoretical guarantee of benefit is provided despite the claim of theoretical analysis.
- Hyperparameters such as alpha and max extra updates were tuned per environment, and results are averaged over only 5 runs.
Experimental setup
Datasets: SMAC Google Research Football SMAC-V2
Benchmarks: SMAC Google Research Football SMAC-V2
Metrics: test win rate training time
Baselines: QMIX QPLEX QATTEN
Builds on: QMIX RND CDS PyMARL2
Closest papers by meaning
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.89)
- Sample-Efficient Multiagent Reinforcement Learning with Reset Replay (ICML 2024 · similarity 0.88)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.88)
- Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing (AAAI 2024 · similarity 0.88)
- Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story (ICML 2025 · similarity 0.88)
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.