Research map / Multi-agent systems and games / Cooperative multi-agent RL
FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FightLadder, an open-source real-time fighting-game platform with pixel observations for competitive MARL, benchmarks PPO-based baselines (IPPO, 2Timescale, FSP, PSRO, League) on Street Fighter, and evaluates them by Elo and exploitability.
Problem
Competitive multi-agent RL lacks a lightweight, open-source benchmark with visual inputs and challenging fighting-game dynamics, along with unified evaluation criteria (Elo, exploitability) for comparing competitive MARL algorithms.
Approach
Wraps classic fighting-game emulators into a Gym-compatible two-player zero-sum environment with pixel observations, a transformed categorical action space, shaped dense rewards, and a unified suite of baselines (IPPO, two-timescale, FSP, PSRO, league training) with Elo and PPO-exploiter evaluation.
Key results
- PPO with curriculum learning defeats all 12 built-in characters with win rate near 1 and completes the full Street Fighter game with over 0.6 win rate.
- League training reaches the highest Elo (1682 left / 1503 right), outperforming IPPO (1082/1164), 2Timescale, FSP, and PSRO (1262/1517), while CPU reference Elo is 1395/1541.
- Exploitability of all two-player baselines remains high (0.88 to 1.00 win rate for a PPO exploiter), indicating none approach Nash equilibrium.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use population-based League training instead of independent learning | PPO in FightLadder · sf_ryu_vs_ryu | Elo: 1082 -> 1682 (Left); 1164 -> 1503 (Right) | Helped |
| Use population-based PSRO instead of independent learning | PPO in FightLadder · sf_ryu_vs_ryu | Elo: 1082 -> 1262 (Left); 1164 -> 1517 (Right) | Helped |
| Train with two-player MARL rather than single-player PPO pretraining | PPO policies in FightLadder · sf_ryu_vs_ryu | exploitability: 0.99 $\pm$ 0.02 -> 0.96 $\pm$ 0.03 (IPPO, Left); 0.99 $\pm$ 0.01 -> 0.91 $\pm$ 0.03 (IPPO, Right) | Helped |
| Add hard-coded special move lists (ablation) | PPO with curriculum learning · sf_ryu_full_game | win rate | No clear effect |
| Use PPO with curriculum learning to train across CPU opponents | PPO · sf_ryu_full_game | win rate: over 0.6 | Helped |
Surprising findings
- Single-player PPO pretrained policies are more exploitable than almost all two-player baseline policies, yet every baseline is still beaten by a PPO exploiter.
- Human players could exploit learned agents with a simple defend-then-low-kick strategy.
Limitations
- Results are mostly on Street Fighter only; other supported games are not benchmarked.
- Scope limited to fully competitive two-player zero-sum settings; multi-player and mixed cooperation-competition not covered.
- Baseline agents remain exploitable by PPO exploiters and human players, so no near-Nash policy is found.
Experimental setup
Datasets: Street Fighter II emulator environment (sf_ryu_vs_ryu, sf_ryu_full_game)
Benchmarks: FightLadder sf_ryu_full_game FightLadder sf_ryu_vs_ryu
Metrics: win rate Elo rating exploitability (win rate of PPO exploiter)
Baselines: IPPO 2Timescale FSP PSRO League training single-agent PPO
Builds on: PPO AlphaStar league training policy space response oracle fictitious self-play Stable-Baselines3
Closest papers by meaning
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.89)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.89)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.88)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.88)
- A Generalist Hanabi Agent (ICLR 2025 · similarity 0.88)
- POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.