SynthCorp: A Platform for Creating Benchmarks for Data Agents
Jiani Zhang, Alon Halevy and colleagues at Google present SynthCorp, a platform that generates enterprise data estates by simulating business events over time and derives verified benchmark tasks for data agents from the simulation log.
Ask this paper
Simulation. Given a scenario and target complexity, SynthCorp writes reusable event types as programs over simulation primitives, schedules routine and disruption events on a virtual calendar, and executes them to fill relational tables, a document store and a data catalog that stay consistent.
Verified gold answers. Every operation is logged, so gold programs for analytical tasks are derived from the log and candidate queries are audited against causal provenance without manual annotation.
Scale. The first release has seven data estates with up to 1,000 operational tables and 256 tasks in five domains, with near-zero error rates on foreign-key, undeclared-write and execution checks.
Signal. Three models across four data-context settings are separated by the benchmark, and multi-event composite tasks still make state-of-the-art agents fail.
Use. Because benchmarks are generated, they can be made harder as agents improve and can supply verifiable rewards for training data agents.
Abstract
Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.