CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
Austin Tudor David Andrews, Jakob Foerster, Rui Ponte Costa and colleagues (Oxford, Google DeepMind, UK AI Security Institute) release CivBench, an open-source benchmark that drives language agents through 300+ turn games of Civilization VI over 76 MCP tools, and report two behavioral failures that are more interesting than the scores.
Ask this paper
Deliberately not a model ranking: With 23 admissible runs the authors say aggregate outcomes do not discriminate models, and pivot to interface-level metrics instead. That restraint is rare and makes the findings more trustworthy.
Agents under-monitor available state: Proactive Monitoring Rate shows agents query victory progress every 30 to 75 turns despite playbook guidance to do it every 20, and in 7 of 20 detectable defeats they never queried inside the 20-turn warning window.
Agents ignore their own plans: RAG@10 measures whether commitments written in a planning reflection get executed within ten turns. It lands between 48.2 and 65.8 percent across models.
Deviation under instruction, not missing capability: Both failures happen with tool access and explicit guidance present, which points at the harness and the reflection loop rather than at model competence.
Why it matters: Everyone building long-horizon agents has seen the agent write a plan and then not follow it. This paper gives that failure a number and an environment to measure it in.
Abstract
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30 to 75 turns, and in 7 of 20 detectable defeats they failed to query within the 20 turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48.2% and 65.8% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://github.com/lmwilki/civ6-mcp