Research map / Large language models / Agents, code and math

Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach

NeurIPS 2024 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TextStarCraft II, a text-based StarCraft II environment for evaluating LLMs in real-time strategy, and a Chain of Summarization (CoS) method that summarizes observations over multiple frames before CoT-based action extraction.

Problem

Existing benchmarks lack evaluation of LLMs on real-time strategic decision-making and long-term planning, and LLMs querying at every time step is too slow for fast-paced games like StarCraft II.

Approach

A text-based SC2 environment (python-sc2 observations converted to text, rule-based micro) plus the Chain of Summarization method, which compresses per-frame observations into single-frame summaries, aggregates K frames into a multi-frame summary, runs CoT reasoning, and extracts K actions into a queue so the LLM is queried less often.

Key results

• Most tested LLMs with full CoS beat the level-5 built-in AI; GPT3.5-Turbo-16k won 11/20 and GPT4 won 12/20 games.
• Fine-tuning on top-25% APU winning games gave a 54/100 win rate versus 28/100 for the full dataset.
• Fine-tuned Qwen 1.8B won 5/10 against a Gold-level human and lost 0/10 to a Grandmaster and a Pro player.

Measured effects reported in the paper

Surprising findings

• Fine-tuning on the full dataset including losing games reduced win rate (28/100) compared with training only on high-APU wins (54/100).
• Llama2 70B without fine-tuning could not follow the task prompts and generate commands.

Limitations

• Relies on rule-based scripts for micro-management and text-only observations, no visual inputs.
• Only the Protoss race is tested as the LLM agent, and only a subset of races is supported.
• Human-AI evaluation uses very few human players (4) and small match counts (10 each).

Experimental setup

Datasets: GPT-3.5-turbo-16k TextStarCraft II interaction logs StarCraft II forum-sourced questions

Benchmarks: TextStarCraft II

Metrics: win rate PBR RUR APU TR

Baselines: GPT-3.5-Turbo-16k GPT-4 Gemini-Pro GLM4 Claude2.1 Llama2 70B AlphaStar StarCraft II built-in AI

Builds on: chain-of-thought ReAct AlphaStar OpenAI Five

Closest papers by meaning

• LLM-PySC2: Starcraft II learning environment for Large Language Models (NeurIPS 2025 · similarity 0.93)
• GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.90)
• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.89)
• GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.89)
• GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps (NeurIPS 2024 · similarity 0.88)
• Can Large Language Models Master Complex Card Games? (NeurIPS 2025 · similarity 0.88)