Research map / Multi-agent systems and games / LLM agents

CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CivBench, an open-source MCP-based long-horizon benchmark on Civilization VI, and uses 23 pilot runs across four model families to measure under-monitoring (PMR) and reflection-action gap (RAG@10).

Problem

Existing agent benchmarks rarely measure whether long-horizon, tool-mediated agents monitor latent state and execute their own stated plans; aggregate outcomes alone are too coarse to reveal these behaviours.

Approach

A Civilization VI environment exposed via 76 MCP tools and a narration layer that only surfaces state when explicitly queried, which lets interface-level metrics (Proactive Monitoring Rate and RAG@K) be computed from traces to separate unavailable information from unqueried information.

Key results

• Across 23 admissible runs, PMR is low (0.96% to 2.13% of non-infrastructure calls), with get_victory_progress queried only every 30-75 turns versus a recommended 20.
• In 7 of 20 detectable defeats, agents made no victory-progress query within the 20-turn warning window.
• RAG@10 ranges from 48.2% to 65.8% across the three model families with sufficient commitments; labeller agreement with a human sample was 92.0% (Cohen's kappa 0.879).

Surprising findings

• Aggregate outcomes (win/loss, normalised score) do not discriminate between models at this sample size (Fisher p=0.488).
• Agents under-query strategically relevant state even when the playbook explicitly recommends checking it every 20 turns.

Limitations

• Small, uneven sample (23 runs, one family with a single run), so between-model differences are descriptive only.
• Shared playbook confounds behaviour with instruction following, and no playbook-free or scripted baseline was viable.
• RAG depends on an LLM-based commitment extraction pipeline and playbook-elicited diaries; results are execution fidelity under this protocol.

Experimental setup

Datasets: CivBench scenarios (Ground Control, Snowflake)

Benchmarks: CivBench

Metrics: Proactive Monitoring Rate (PMR) RAG@10 normalised score ICC missed-warning rate

Baselines: CivRealm CivAgent Vox Deorum Civilization VI built-in AI

Builds on: Model Context Protocol Inspect framework CivRealm

Closest papers by meaning

• T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning (NeurIPS 2025 · similarity 0.88)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.88)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
• CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents (ICLR 2024 · similarity 0.88)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.88)