Research map / Multi-agent systems and games / Human-AI and goal-conditioned
CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CivRealm, a Freeciv-based (Civilization) multi-agent environment with tensor and language APIs, 10 mini-game types, and baseline RL, BaseLang and Mastaba LLM agents for evaluating learning and reasoning.
Problem
Existing interactive environments emphasize learning over reasoning and have static state/action spaces; decision-making agents need a testbed combining learning from experience with reasoning in open-ended, changing, multi-player settings.
Approach
A Civilization-inspired environment built on Freeciv with imperfect information, general-sum play, changing player counts, and dynamic state/action spaces, offering both tensor (RL) and language (LLM) interfaces plus auto-generated mini-games; baselines include an AlphaStar-inspired PPO agent and LLM agents BaseLang (AutoGPT-like) and hierarchical Mastaba.
Key results
- RL agent reaches ~90% success on CityTileWonder, while most minitasks remain hard for the tensor baseline.
- In full games the RL agent mainly optimizes unit production, a myopic strategy, with score improving over 2 million training steps.
- Mastaba outperforms BaseLang in development pace (population, military units, score) and reaches a larger explored and settled area by turn 113 versus BaseLang's turn 210.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a hierarchical advisor-and-worker structure instead of BaseLang's local-agent structure | Mastaba · CivRealm full game | score | Helped |
| use a hierarchical advisor-and-worker structure instead of BaseLang's local-agent structure | Mastaba · CivRealm full game | explored and settled area: by turn 113; BaseLang's progress by turn 210 | Helped |
| use a hierarchical advisor-and-worker structure instead of BaseLang's local-agent structure | Mastaba · CivRealm full game | survival duration | Mixed |
| train on tasks with immediate rewards and relatively short-term planning | Tensor-based RL · CivRealm mini-games | learning capability | Helped |
| apply tensor-based RL to tasks with sparse, delayed rewards and long-term strategic planning | Tensor-based RL · CivRealm mini-games | ability to identify viable solutions | Hurt |
| prioritize unit production to increase score | Tensor-based RL · CivRealm full game | score | Helped |
Surprising findings
- In the SettlerBuildCity task, the RL method exhibited shortcut learning at the hard level.
- Mastaba's global advisor did not prevent failure to defend against pirate invasions, showing LLM difficulty in prioritizing critical issues.
Limitations
- Both RL and LLM agents struggle to make substantial progress in the full game.
- LLM experiments use only GPT-3.5-turbo and a small evaluation (10 maps, 5 games each).
Experimental setup
Datasets: CivRealm mini-games (procedurally generated)
Benchmarks: CivRealm mini-games CivRealm full game
Metrics: success rate aggregated game score population technologies researched units built/killed explored and settled area
Baselines: BaseLang Mastaba tensor-based RL (AlphaStar-inspired PPO)
Builds on: AlphaStar PPO AutoGPT Freeciv
Closest papers by meaning
- CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI (NeurIPS 2026 · similarity 0.88)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
- LLM-PySC2: Starcraft II learning environment for Large Language Models (NeurIPS 2025 · similarity 0.86)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.86)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.86)
- Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.