Research map / Multi-agent systems and games / Cooperative multi-agent RL
Learning Progress Driven Multi-Agent Curriculum
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPMARL, which uses a TD-error-based learning progress objective within self-paced RL to automatically control the number of agents as a MARL curriculum, outperforming reward-based and heuristic curricula on sparse-reward MPE, XOR, and SMAC-v2 tasks.
Problem
Automatic curriculum over the number of agents in sparse-reward MARL is hampered by high-variance episode-return task difficulty estimates and by credit-assignment difficulty that grows when more agents yield higher returns.
Approach
Replace the expected episode return in self-paced RL with a learning-progress measure given by the expected TD (value) loss of MAPPO's centralized critic, averaged over all state transitions, and let the context distribution over number of agents move from an initial distribution to the target task.
Key results
- On Simple-Spread, SPMARL reaches 33.36 mean evaluation return vs 27.48 for SPRLM and 24.65 for ALPGMM during training.
- On the 20-player XOR game, SPMARL achieves 143.08 mean return vs 141.45 for SPRLM and 104.80 for ALPGMM, with faster convergence.
- On SMAC-v2 Protoss 8v8, SPMARL achieves 0.31 mean win rate vs 0.11 for SPRLM and 0.28 for ALPGMM, and it shows lower std of TD-error than episode return.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace reward-based curriculum updates with a TD-error learning-progress objective | SPMARL vs SPRLM · SimpleSpread | evaluation performance: 27.48 (5.38) -> 33.36 (1.84) | Helped |
| Replace reward-based curriculum updates with a TD-error learning-progress objective | SPMARL vs SPRLM · XOR | evaluation performance: 141.45 (16.98) -> 143.08 (7.72) | No clear effect |
| Replace reward-based curriculum updates with a TD-error learning-progress objective | SPMARL vs SPRLM · SMAC-v2 Protoss 6v6 | evaluation performance: 0.13 (0.15) -> 0.45 (0.01) | Helped |
| Replace reward-based curriculum updates with a TD-error learning-progress objective | SPMARL vs SPRLM · SMAC-v2 Protoss 7v7 | evaluation performance: 0.08 (0.11) -> 0.38 (0.02) | Helped |
| Replace reward-based curriculum updates with a TD-error learning-progress objective | SPMARL vs SPRLM · SMAC-v2 Protoss 8v8 | evaluation performance: 0.11 (0.07) -> 0.31 (0.01) | Helped |
| Add an agent-number curriculum instead of training directly on the target task | MAPPO · SimpleSpread, XOR, SMAC-v2 Protoss 5v5/6v6/7v7/8v8 | evaluation performance: 0.00 (0.00) | Helped |
| Use a linear curriculum from fewer to more agents | MAPPO · SimpleSpread | evaluation performance: 0.00 (0.00) | No clear effect |
| Use a linear curriculum from fewer to more agents | MAPPO · XOR | evaluation performance: 26.80 (1.16) | Helped |
| Set the performance threshold V_LB to different values (ablation) | SPMARL · SMAC-v2 tasks | performance | No clear effect |
| Use a learning-progress objective based on TD error rather than episode return | SPMARL vs SPRLM · SMAC-v2 Protoss 7v7 and 8v8 | objective estimation standard deviation | Helped |
Surprising findings
- SPRLM fails to generate an effective curriculum on most SMAC-v2 tasks except 5v5, attributed to the sparse {-1,0,1} reward setting.
- The Linear increasing curriculum completely fails on Simple-Spread, since few agents rarely receive non-zero sparse rewards.
Limitations
- The KL constraint from SPRL may be unnecessary with the learning-progress objective, left unverified.
- V_LB threshold is set heuristically (about 80% of target task performance) and requires tuning.
- Gains over ALPGMM are not significant on some SMAC-v2 tasks (e.g., 5v5 and 6v6 p-values above 0.05).
Experimental setup
Datasets: MPE Simple-Spread (modified sparse reward) XOR matrix game (20 players) SMAC-v2 Protoss 5v5-8v8
Benchmarks: MPE Simple-Spread XOR matrix game SMAC-v2 Protoss SMAC-v2 Terran SMAC-v2 Zerg BenchMARL Balance BenchMARL Wheel
Metrics: evaluation episode return on target task standard deviation of TD error vs episode return Mann-Whitney U test p-values
Baselines: SPRLM Linear curriculum ALPGMM VACL W/O teacher
Builds on: self-paced reinforcement learning MAPPO variational automatic curriculum learning contextual reinforcement learning
Closest papers by meaning
- Automated curriculum generation through setter-solver interactions (ICLR 2020 · similarity 0.89)
- PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (NeurIPS 2021 · similarity 0.88)
- Automatic Curriculum Learning through Value Disagreement (NeurIPS 2020 · similarity 0.88)
- Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (ICLR 2020 · similarity 0.88)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.