Research map / Multi-agent systems and games / LLM agents
LLM-PySC2: Starcraft II learning environment for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLM-PySC2, a StarCraft II environment giving LLMs the full pysc2 action space, multimodal and wiki observations, and a native asynchronous multi-agent communication framework, then benchmarks mainstream LLMs on full games and micro tasks.
Problem
LLMs cannot interface with StarCraft II's hundreds of pysc2 actions and lack native multi-agent support; existing LLM StarCraft environments use oversimplified observation and action spaces.
Approach
Expose the complete pysc2 action space as text actions parsed by regular expressions, provide multimodal and wiki-knowledge observations, and run each LLM agent in independent asynchronous threads with natural-language inter-agent messaging so query latency stays constant as agent count grows.
Key results
- In complete games on Simple64, GPT-4o-mini with easy control and easy build won 100% at level-1 and 2 but 0% at level-7; standard control modes dropped to 0% by level-3.
- In LLM-SMAC tasks, all tested LLMs performed poorly; the best win rate was 25% (GLM-4-plus on 2s3z) and most scenarios were 0%.
- In LLM-PySC2 level-1 tasks, Claude 3 Haiku reached 90% win rate on task1 and GPT-4o reached 100% on task3, while Llama 3.1 405B did not outperform 70B clearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard unit control instead of easy control | GPT-4o-mini in complete Simple64 games with Easy Build · Simple64 level-3 | winning rate: 92% -> 0% | Hurt |
| Use standard building instead of easy building | GPT-4o-mini in complete Simple64 games with Easy Control · Simple64 level-3 | winning rate: 92% -> 0% | Hurt |
| Increase task difficulty from level-1 to level-3 | GPT-3.5-turbo · LLM-PySC2 tasks | winning rate: 58% -> 0% | Hurt |
| Scale Llama3.1 from 70B to 405B parameters | Llama3.1 on LLM-PySC2 · LLM-PySC2 tasks | decision-making ability | No clear effect |
| Use GPT-o1-mini reasoning model rather than non-reasoning models | LLM-PySC2 · LLM-PySC2 tasks | decision-making ability | No clear effect |
Surprising findings
- Llama 3.1 405B did not significantly outperform Llama 3.1 70B, suggesting scaling does not straightforwardly help decision-making.
- Reasoning model o1-mini did not significantly improve decision-making over non-reasoning models in this unseen environment.
Limitations
- Most evaluations use only API-hosted models with small numbers of repeated games (about 12 to 20 per setting).
- Standard control and build modes show severe performance drops due to hallucinations, invalid actions, and poor coordination.
- No training or learning method is proposed; results are zero-shot prompting only.
Experimental setup
Datasets: SC2LE SMAC maps StarCraft II game Wiki knowledge
Benchmarks: SMAC (LLM-SMAC tasks) LLM-PySC2 tasks (8 new micro scenarios) complete StarCraft II games (Simple64, levels 1-7)
Metrics: winning rate kill/death ratio
Baselines: TextStarCraft II Swarm Brain
Builds on: SC2LE SMAC TextStarCraft II
Closest papers by meaning
- Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach (NeurIPS 2024 · similarity 0.93)
- TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.89)
- Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs (NeurIPS 2025 · similarity 0.88)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.