Research map / Multi-agent systems and games / Cooperative multi-agent RL
FoX: Formation-Aware Exploration in Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FoX, a formation-aware exploration framework for cooperative MARL that defines formation-based equivalence over observation differences, combines count-based formation exploration with a VAE-style formation-awareness intrinsic reward (with gradient flipping), and built on QMIX.
Problem
Exploration in cooperative MARL is hard due to partial observability and an exploration space that grows exponentially with the number of agents, especially under sparse rewards.
Approach
Define a formation (pairwise observation-difference distances and SimHash angles) and an equivalence relation over joint observations so agents explore diverse formations rather than the full observation product space; add a variational formation-awareness reward so each agent can infer formations it belongs to from its own observations.
Key results
- FoX significantly outperforms baselines on sparse SMAC across 6 scenarios, with notable gains on 2m_vs_1z where other baselines struggled to learn.
- On GRF, FoX outperforms QPLEX, CDS, MAVEN, QMIX and visitation-count variants across four academy scenarios.
- Ablations show removing the intrinsic rewards, Q^F, shared Q, or gradient flipping each reduces performance on GRF 3_vs_1_with_keeper; beta1=0.01 and beta2=0.01 were best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FoX formation-aware exploration instead of baseline MARL algorithms | QMIX-based MARL · Sparse SMAC: 8m, 2s3z, 2m_vs_1z, 3s5z, 1c3s5z | performance | Helped |
| Use FoX formation-aware exploration instead of baseline MARL algorithms | MARL · GRF Academy scenarios | performance | Helped |
| Use formation-based visitation counts instead of joint-observation visitation counts | QMIX · GRF Academy scenarios | performance | Helped |
| Add formation-based exploration reward (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
| Add formation-awareness intrinsic reward (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
| Add formation-information Q-function Q^F (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
| Add shared Q learning (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
| Add gradient flipping to remove irrelevant trajectory information from the latent variable (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
| Set exploration-reward weight β1 to 0.01 rather than 0 (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | learning performance | Helped |
| Increase exploration-reward weight β1 above 0.01 (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | learning speed | Hurt |
| Increase formation-awareness reward weight β2 above 0.01 (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | learning performance | Helped |
| Select F^{i,max,min} instead of F^{all} (ablation) | FoX · GRF Academy_3_vs_1_with_keeper | performance | Helped |
Surprising findings
- Formation selection matters: using all agents (F^all) decreased performance relative to the max/min formation set despite holding similar information.
- In the 3m scenario, EMC performs comparably to FoX, the only case where a baseline matches it.
Limitations
- Formation computation is ineffective when observations of dead or out-of-sight agents are zeroed or constant, relying on environment-specific observation behavior.
- Hyperparameters beta1, beta2 and formation index set F^i must be tuned per environment.
Experimental setup
Datasets: SMAC (sparse variant) Google Research Football academy scenarios
Benchmarks: SMAC sparse (3m, 8m, 2s3z, 2m_vs_1z, 3s5z, 1c3s5z) GRF Academy_3_vs_1_with_keeper GRF Academy_4_vs_2_with_keeper GRF Academy_counterattack_hard GRF Academy_corner
Metrics: win rate average reward
Baselines: EMC MASER ROMA MAVEN LIIR COMA QPLEX CDS QMIX QMIX+JC QMIX+IC
Builds on: QMIX EMC count-based exploration variational autoencoder gradient flipping
Closest papers by meaning
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.86)
- Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.85)
- Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing (AAAI 2024 · similarity 0.85)
- Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.