Research map / Multi-agent systems and games / Cooperative multi-agent RL
Learning Distinguishable Trajectory Representation with Contrastive Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Trajectory Representation (CTR), which uses learnable agent identity vectors and a multi-agent InfoNCE-style contrastive loss on agents' GRU-encoded trajectory representations to encourage diversity in parameter-shared MARL, implemented on QMIX and MAPPO.
Problem
Parameter-sharing in cooperative MARL makes agents homogeneous, and prior mutual-information-with-identity methods (variational inference) overfit to known trajectories, hurting exploration.
Approach
Replace fixed one-hot agent identities and variational MI bounds with learnable identity vectors trained via a multi-agent contrastive loss (|A| positive pairs, |A|(|A|-1) negatives) over trajectory representations from an encoder plus GRU, added as an auxiliary loss to the MARL objective.
Key results
- CTR+QMIX reaches 0.85 on corridor and 0.83 on 3s5z_vs_3s6z versus 0.36 and 0.68 for QMIX... (Table 1: QMIX corridor 0.36, 3s5z_vs_3s6z 0.68)
- CTR+QMIX achieves 0.85±0.04 on terran_5_vs_5 vs 0.68 for QMIX, and 0.82 vs 0.11 for QMIX on terran_20_vs_20 in the scalability study
- Ablation: replacing learnable identity with one-hot performs near QMIX, and TC loss underperforms the multi-agent contrastive loss
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CTR to QMIX | QMIX · Pac-Men; SMAC; SMACv2 | average return: Pac-Men: 0.21±0.04 -> 0.78±0.04; 3s5z: 0.72±0.13 -> 0.95±0.03; 2c_vs_64zg: 0.85±0.08 -> 0.87±0.03; 7sz: 0.17±0.02 -> 0.8 | Helped |
| Replace the GRU autoregressive model with MLPs (ablation) | CTR+QMIX · SMAC 6h_vs_8z | average return | Hurt |
| Replace learnable identity representations with fixed one-hot vectors (ablation) | CTR+QMIX · SMAC 3s5z; 2c_vs_64zg; corridor | average return | Hurt |
| Use trajectory-classification loss instead of the multi-agent contrastive loss (ablation) | CTR+QMIX · SMAC 3s5z; 2c_vs_64zg; corridor | average return | Hurt |
| Use vanilla InfoNCE instead of the loss with |A| positive samples (ablation) | CTR+QMIX · SMAC corridor | average return | Hurt |
| Use vanilla InfoNCE instead of the loss with |A| positive samples (ablation) | CTR+QMIX · SMAC 3s5z; 2c_vs_64zg | average return | Helped |
| Replace the GRU autoregressive model with MLPs (ablation) | CTR+QMIX · SMAC 3s5z; 2c_vs_64zg | average return | No clear effect |
Surprising findings
- Mutual-information-based methods such as EOI perform poorly on SMACv2 and in super-hard SMAC scenarios despite diversity objectives
- CTR-With-InfoNCE (single-positive contrastive loss) shows obvious degradation in the corridor scenario compared with the multi-positive loss
Limitations
- Requires centralized training to collect all agents' trajectories, so not applicable to fully decentralized MARL
- Number of negative samples limited when the number of agents is small
- Does not automatically control diversity versus homogeneous behavior
Experimental setup
Datasets: SMAC SMACv2 Pac-Men (custom grid world)
Benchmarks: SMAC SMACv2 Pac-Men
Metrics: test win rate average return visitation heatmaps t-SNE of trajectory representations
Baselines: QMIX QTRAN MAVEN EOI SCDS PMIC LIPO FoX MAPPO
Builds on: QMIX InfoNCE MAPPO EOI MAVEN
Closest papers by meaning
- Trajectory-Class-Aware Multi-Agent Reinforcement Learning (ICLR 2025 · similarity 0.87)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.86)
- Acting Beyond Learning: Imagination-Assisted Decision-Making in the Visual-based Multi-Agent Cooperative Scenarios (AAAI 2025 · similarity 0.86)
- Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion (NeurIPS 2024 · similarity 0.85)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.