Research map / Multi-agent systems and games / LLM agents
The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the PokéAgent Challenge, a two-track benchmark (competitive Pokémon battling and RPG speedrunning) with a 20M+ battle dataset, RL/LLM/heuristic baselines, and results from a NeurIPS 2025 competition.
Problem
Existing benchmarks rarely combine adversarial partial observability with long-horizon planning, and prior Pokémon AI efforts lacked standardized evaluation, leaderboards, and fair cross-paradigm comparison.
Approach
A dual-track living benchmark built on Pokémon Showdown battling and Pokémon Emerald speedrunning, with standardized evaluation (FH-BT rating, milestone-based speedrun scoring), large human/self-play datasets, open-source harness baselines, and an analysis showing battling is near-orthogonal to standard LLM benchmarks.
Key results
- Heatz completed the Speedrunning route to Roxanne in 40:13 using Scripted Policy Distillation, nearly twice as fast as the second-place pure RL team.
- Across 16 models overlapping BenchPress, no standard benchmark correlates strongly with GXE (max Spearman rho 0.77; mean |rho| 0.45).
- Without a harness, raw frontier VLMs achieve effectively 0% task completion on the speedrunning track.
Surprising findings
- Pokémon battling is nearly orthogonal to standard LLM benchmarks; the rank-2 SVD explaining 91% of standard benchmark variance captures only 27% of GXE variance.
- CLI coding agents (Claude Code, Codex CLI, Gemini CLI) fail to maintain coherence on speedrunning despite using the same frontier models that succeed with the domain harness.
Limitations
- Speedrunning scoped only to the first gym, not the full game.
- Current RL upper bound on battling remains less than superhuman.
- Gen 1 OU baseline performance is an outlier relative to human ladder ratings.
Experimental setup
Datasets: Pokémon Showdown replays Pokémon Showdown self-play battles Smogon usage statistics
Benchmarks: PokéAgent Battling Track (Gen 1 OU, Gen 9 OU) PokéAgent Speedrunning Track (Pokémon Emerald first gym) BenchPress matrix
Metrics: FH-BT rating GXE Glicko-1 completion percentage completion time action count
Baselines: PokéChamp Metamon RL policies Claude Code Codex CLI Gemini CLI FoulPlay Heatz
Builds on: PokéChamp Metamon Glicko-1 Bradley-Terry model BenchPress
Closest papers by meaning
- PokéChamp: an Expert-level Minimax Language Agent (ICML 2025 · similarity 0.91)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.85)
- MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.85)
- TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.84)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.