Research map / Multi-agent systems and games / Cooperative multi-agent RL
PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PORTAL, an automatic curriculum learning framework for MARL that selects intermediate SMAC tasks by difficulty (average return) and similarity (GMM KL divergence over a shared entity-relation feature space) to speed up learning of hard final tasks.
Problem
MARL algorithms struggle to solve complex cooperative multiagent tasks from scratch due to sample inefficiency; existing multiagent curriculum methods assume all intermediate tasks help the final task or rely on hand-crafted task contexts.
Approach
Select curriculum tasks with two criteria: moderate difficulty measured by the pretrained policy's average return (jumpstart performance), and task similarity to the final task measured by KL divergence between GMMs fit to observations; plus a relation-weighted entity feature network that yields a shared observation feature space enabling policy transfer across tasks with different numbers of enti
Key results
- PORTAL achieves the best test win rate over baselines on intermediate and final tasks across Marines, S&Z and MMM curricula (Figure 3).
- Relation aggregation gives 0.5587 win rate on 7m vs 9m under transfer versus 0.2182 for sum aggregation and 0.1726 for mean (Table 2).
- Policy transfer performance on the final task increases with GMM-based similarity between the candidate task and the final task (Figure 6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use relation-weighted feature aggregation instead of sum aggregation during curriculum transfer (ablation) | PORTAL with HPN-VDN · 8m vs 10m | test win rate: 0.8422(±0.0811) -> 0.9074(±0.0736) | Helped |
| Use relation-weighted feature aggregation instead of sum aggregation during curriculum transfer (ablation) | PORTAL with HPN-VDN · 7m vs 9m | test win rate: 0.2182(±0.0702) -> 0.5587(±0.0953) | Helped |
| Use relation-weighted feature aggregation instead of mean aggregation during curriculum transfer (ablation) | PORTAL with HPN-VDN · 8m vs 10m | test win rate: 0.8207(±0.1239) -> 0.9074(±0.0736) | Helped |
| Use relation-weighted feature aggregation instead of mean aggregation during curriculum transfer (ablation) | PORTAL with HPN-VDN · 7m vs 9m | test win rate: 0.1726(±0.0504) -> 0.5587(±0.0953) | Helped |
| Use relation-weighted feature aggregation instead of mean aggregation when training from scratch (ablation) | PORTAL with HPN-VDN · 8m vs 10m | test win rate: 0.7869(±0.0742) -> 0.8455(±0.0689) | Helped |
| Use relation-weighted feature aggregation instead of mean aggregation when training from scratch (ablation) | PORTAL with HPN-VDN · 7m vs 9m | test win rate: 0.1532(±0.0434) -> 0.2186(±0.0592) | Helped |
| Use relation-weighted feature aggregation instead of sum aggregation when training from scratch (ablation) | PORTAL with HPN-VDN · 8m vs 10m | test win rate: 0.8540(±0.0814) -> 0.8455(±0.0689) | No clear effect |
| Use relation-weighted feature aggregation instead of sum aggregation when training from scratch (ablation) | PORTAL with HPN-VDN · 7m vs 9m | test win rate: 0.2255(±0.0553) -> 0.2186(±0.0592) | No clear effect |
| Select curricula using task similarity in addition to task difficulty | PORTAL · SMAC task series | test win rate | Helped |
| Use 8m vs 10m rather than 20m or 25m as the intermediate task before 7m vs 9m | PORTAL curriculum · 7m vs 9m | test win rate | Helped |
Surprising findings
- Using task difficulty alone (ERFN-MAACL) is not good enough; selecting the most difficult or same-size task can hurt versus a similar intermediate task.
- Mean aggregation performs worst in both training and transfer because it ignores entity count information.
Limitations
- Requires a given set of candidate tasks; cannot generate tasks automatically.
- Difficulty thresholds Rmin and Rmax and the GMM hyperparameters must be set; performance evaluated only on SMAC.
Experimental setup
Datasets: SMAC Marines task series (5m to 25m, 7m vs 9m, 8m vs 10m) SMAC stalkers and zealots series (2s3z to 3s5z vs 4s8z) SMAC medivac marauders marines series (MMM0 to MMM10)
Benchmarks: SMAC
Metrics: test win rate convergence test win rate t-SNE visualization
Baselines: HPN-QMIX DYMA OO MAACL ERFN-OO ERFN-MAACL
Builds on: VDN HPN-VDN DYMA MAACL SMAC
Closest papers by meaning
- TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL (ICML 2021 · similarity 0.90)
- Automated curriculum generation through setter-solver interactions (ICLR 2020 · similarity 0.90)
- Learning Progress Driven Multi-Agent Curriculum (ICML 2025 · similarity 0.89)
- Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs (NeurIPS 2025 · similarity 0.86)
- Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning (AAAI 2024 · similarity 0.86)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.