Research map / Reinforcement learning / Visual and representation RL
Sample-Efficient Multiagent Reinforcement Learning with Reset Replay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARR, which makes off-policy MARL sample-efficient in parallel environments by training at a high replay ratio (e.g., 50) with Shrink & Perturb resets and random amplitude scale augmentation.
Problem
Off-policy multiagent RL has low sample efficiency, and prior work has not enabled high replay ratio training for MARL, especially in parallel environment settings, where plasticity loss hampers high-replay training.
Approach
Combine a high replay ratio (updates per environment interaction) with periodic Shrink & Perturb parameter resets for the agent and centralized critic networks to maintain plasticity, plus random amplitude scale data augmentation to diversify repeated batches; plugs into existing off-policy MARL algorithms with slight modification.
Key results
- MARR at replay ratio 50 substantially improves QMIX, QPLEX and ATM win rates on 7 SMAC maps within 1M environment steps (8 parallel environments).
- On MPE predator-prey tasks, MARR greatly improves MADDPG and FACMAC on 3a and 6a/9a for FACMAC, but fails to boost MADDPG on 6a and 9a.
- Ablation shows removing Shrink & Perturb degrades performance significantly, while random amplitude scale adds further gains only when Shrink & Perturb is used.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MARR (Shrink & Perturb reset plus random amplitude scale) with replay ratio 50 to QMIX, QPLEX, ATM | off-policy SMAC MARL algorithms · SMAC (7 maps) | test win rate at 1M steps: large margin on 5 maps; ATM only on 3s5z vs 3s6z and corridor | Helped |
| Add MARR (replay ratio 25) to MADDPG and FACMAC | MADDPG, FACMAC · MPE predator-prey 3a, 6a, 9a | episodic return | Mixed |
| Remove Shrink & Perturb from MARR (keep random amplitude scale) at replay ratio 50 (ablation) | MARR-QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate: performance degenerates significantly in both scenarios | Hurt |
| Remove random amplitude scale from MARR (keep Shrink & Perturb) (ablation) | MARR-QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate: smaller drop than removing Shrink & Perturb | Hurt |
| Random amplitude scale alone (without Shrink & Perturb) (ablation) | QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate | No clear effect |
| Increase replay ratio from 50 to 100 in MARR-QMIX (ablation) | MARR-QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate | Hurt |
| Use parallel environments (8) instead of series sampling for QMIX at high replay ratio (ablation) | QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate | Helped |
| Replace Shrink & Perturb with weight decay, L2 Init, MAP, or layer norm (ablation) | MARR-QMIX · SMAC 5m vs 6m and 10m vs 11m | test win rate | Hurt |
| Apply Shrink & Perturb reset during high-replay training (ablation) | QMIX at replay ratio 50 · SMAC 5m vs 6m and 10m vs 11m | L2 gap to initial network | Helped |
Surprising findings
- Replay ratios of 100 hurt MARR performance, indicating overfitting even with resets.
- MARR fails to boost MADDPG on the 6a and 9a MPE tasks within the step budget.
Limitations
- Optimal replay ratio varies by scenario and requires tuning; NRR=100 degrades performance.
- MARR cannot guarantee gains on tasks that are hard for the base algorithm (e.g., MADDPG on 6a/9a).
- Evaluated only on SMAC and MPE predator-prey with off-policy algorithms.
Experimental setup
Datasets: SMAC MPE predator-prey
Benchmarks: SMAC MPE
Metrics: test win rate episodic return L2 gap to initial parameters
Baselines: QMIX QPLEX ATM MADDPG FACMAC L2 Init weight decay layer normalization multiagent permutation
Builds on: Shrink & Perturb random amplitude scale QMIX QPLEX MADDPG
Closest papers by meaning
- Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.86)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.86)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.