Research map / Multi-agent systems and games / Cooperative multi-agent RL
ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ZSC-Eval, a toolkit and benchmark for evaluating zero-shot coordination agents using behavior-preferring reward partners selected by best-response diversity and scored with the BR-Prox metric, tested in Overcooked and Google Research Football.
Problem
Evaluating zero-shot coordination (ZSC) agents is unreliable because evaluation partners do not match deployment-time partner distributions and mean episode returns ignore generalization gaps and unequal partner capabilities.
Approach
Generate evaluation partner candidates from event-based behavior-preferring rewards, select a diverse subset by maximizing the determinant-based diversity of their approximate best responses (BR-Div, via DPP sampling), and measure ego agents with BR-Prox, the ratio of ego return to approximate best-response return over partner combinations.
Key results
- ZSC-Eval rankings of ZSC algorithms in Coord. Ring and Counter Circ. layouts match human rankings with Spearman correlation 0.90 and 1.00 (Counter Circ.), versus 0.60 and 0.10 for trained self-play agents as partners in Counter Circ.
- In GRF, BR-Prox for SP is 0.20 while E3T, FCP, MEP, TrajeDi, COLE and HSP range 0.66 to 0.81.
- Human study with 145 valid participants ranked 7 agents across 2 Overcooked layouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select evaluation partners by BR-Div instead of P-Div | ZSC-Eval · Overcooked evaluation-partner selection | Population diversity of best responses | Helped |
| Evaluate with ZSC-Eval partners rather than trained self-play partners | ZSC algorithm rankings · Overcooked Coord. Ring | Spearman rank correlation with human evaluation: 0.70 -> 0.90 | Helped |
| Evaluate with ZSC-Eval partners rather than human proxy partners | ZSC algorithm rankings · Overcooked Counter Circ. | Spearman rank correlation with human evaluation: 0.60 -> 1.00 | Helped |
| Increase coordination complexity through more frequent agent interaction | Overcooked testbeds · Bothway Coord. and Blocked Corr. | Performance differentiation among ZSC algorithms | Helped |
| Add the multi-recipe mechanism to increase task complexity | Overcooked testbeds · Multi-recipe layouts | Performance differentiation among ZSC algorithms | Helped |
| Increase population size | Population-based ZSC algorithms · Overcooked | BR-Prox performance | Helped |
| Increase population size | Population-based ZSC algorithms · Overcooked multi-recipe layouts | BR-Prox performance with expert evaluation partners | No clear effect |
| Use expert rather than moderate evaluation partners | Current ZSC algorithms · Overcooked | BR-Prox performance | Hurt |
Surprising findings
- Current ZSC algorithms perform worse with expert evaluation partners than with moderate partners, and larger population sizes help less against experts.
- Commonly used Overcooked layouts (Forced Coord., Asymm. Adv.) fail to differentiate algorithm performance, with self-play performing well.
Limitations
- Event-based reward design is handcrafted and may not fully represent deployment-time partners.
- Human study and benchmarks are limited to Overcooked (two layouts for human study) and one GRF scenario.
Experimental setup
Datasets: Overcooked-AI layouts Google Research Football 3 vs 1 with Keeper
Benchmarks: Overcooked Google Research Football 3 vs 1 with Keeper
Metrics: BR-Prox Spearman rank correlation population diversity of best responses 95% confidence intervals
Baselines: FCP MEP TrajeDi HSP COLE E3T Self-play
Builds on: Fictitious Co-Play Hidden-utility Self-Play Determinantal Point Process PPO Overcooked-AI
Closest papers by meaning
- Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination (ICML 2025 · similarity 0.89)
- Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games (AAAI 2026 · similarity 0.89)
- OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination (ICLR 2025 · similarity 0.89)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
- Learning to Cooperate with Humans using Generative Agents (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.