Research map / Multi-agent systems and games / Cooperative multi-agent RL
Trajectory-Class-Aware Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRAMA, which clusters trajectories in a quantized (modified VQ-VAE) latent space and trains agents to predict and condition on their trajectory class for versatile multi-task MARL on SMACv2-style tasks.
Problem
Generalizing multi-agent reinforcement learning policies across multiple tasks with different joint policies and coordination when task IDs are unknown during training and execution.
Approach
Agents infer a trajectory class from partial observations and condition their action policy on a learned class representation; trajectory classes come from K-means over VQ-VAE-quantized trajectory embeddings, with a modified coverage loss indexed by timestep and class and centroid initialization for stable labels.
Key results
- On the multi-task problems, TRAMA reports consistently higher mean return than QMIX, QPLEX, RODE, LDSA, MASER, EMC and LAGMA across SurComb3, SurComb4, reSurComb3 and reSurComb4.
- Ablation on SurComb4 shows removing centroid initialization makes class labels change frequently and degrades performance; removing the J(t,k) coverage loss lowers prediction accuracy.
- On reSurComb4 with n_cl=4, agents' trajectory-class prediction accuracy is reported as high throughout episodes (exact number shown only in figure).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use trajectory-class-aware coverage indexing J(t,k) instead of timestep-only J(t) (ablation) | TRAMA · SurComb4 | mean return | Helped |
| Use centroid initialization for clustering rather than No-Init (ablation) | TRAMA · SurComb4 | mean return | Helped |
| Use centroid initialization for clustering rather than No-Init (ablation) | TRAMA · SurComb4 | trajectory-class prediction accuracy | Helped |
| Use trajectory-class-aware coverage indexing J(t,k) instead of J(t), retaining centroid initialization (ablation) | TRAMA · SurComb4 | trajectory-class prediction accuracy | Helped |
| Use a learned trajectory-class representation instead of a one-hot class vector (ablation) | TRAMA · SurComb3 and reSurComb4 | mean return | Helped |
| Use a learned trajectory-class representation instead of a one-hot class vector (ablation) | TRAMA · p5_vs_5 and t5_vs_5 | mean return | Helped |
| Use TRAMA rather than baseline MARL methods | Multi-task problems built on SMACv2 · SurComb3, reSurComb3, SurComb4, and reSurComb4 | mean return | Helped |
| Use TRAMA rather than baseline MARL methods | SMACv2 · p5_vs_5 and t5_vs_5 | mean return | Helped |
| Use TRAMA rather than baseline MARL methods | SMAC single-task benchmarks · MMM2 and 6h_vs_8z | mean return | Helped |
| Use a larger number of trajectory classes | TRAMA · Multi-task settings | trajectory-class prediction accuracy | Hurt |
Surprising findings
- Win rate gaps between TRAMA and baselines are not distinct on multi-task problems, so the authors use mean return instead, arguing high win rate can reflect task specialization.
Limitations
- Number of trajectory classes n_cl must be set manually and affects performance; adaptive selection only sketched via silhouette score.
- Evaluated only on StarCraft II (SMAC/SMACv2) variants; the multi-task problems are custom and built from a few unit combinations.
Experimental setup
Datasets: SMACv2 p5_vs_5 SMACv2 t5_vs_5 SMAC 1c3s5z SMAC 5m_vs_6m SMAC MMM2 SMAC 6h_vs_8z custom multi-task SurComb3/reSurComb3/SurComb4/reSurComb4
Benchmarks: SMACv2 p5_vs_5 SMACv2 t5_vs_5 SMAC 1c3s5z SMAC 5m_vs_6m SMAC MMM2 SMAC 6h_vs_8z SurComb3 reSurComb3 SurComb4 reSurComb4
Metrics: mean return win rate silhouette score trajectory-class prediction accuracy
Baselines: QMIX QPLEX RODE LDSA MASER EMC LAGMA Updet RiskQ
Builds on: QPLEX LAGMA VQ-VAE PyMARL SMACv2
Closest papers by meaning
- Toward Efficient Multi-Agent Exploration With Trajectory Entropy Maximization (ICLR 2025 · similarity 0.88)
- Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (ICML 2018 · similarity 0.88)
- Learning Distinguishable Trajectory Representation with Contrastive Loss (NeurIPS 2024 · similarity 0.87)
- Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning (AAAI 2023 · similarity 0.87)
- LAGMA: LAtent Goal-guided Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.