Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors

Engineers at Meta present Benchmark Factory, a meta-agent loop that builds coding-agent benchmarks from an organization's own merged code changes and checks them with quality anchors that need no per-task human labels.
Ask this paper
Loop. For each candidate benchmark the agent writes a sourcing query over landed code changes, runs models of known relative strength, checks the anchors, and either accepts the benchmark, repairs it, or hands it to a human after K rounds. A fresh context is used each iteration and state lives in a file.
Anchors. The anchors are a-priori checks such as capability ordering with a margin, a difficulty band and run-to-run variance. The grader and the expected ordering are frozen, so the loop cannot repair its way past them.
Out-of-sample checks. On configurations never used to build them, the benchmarks ranked Claude Opus 4.7 at high reasoning effort above low effort (60.6% vs 54.4%, p=0.002) and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%, p=0.004). 9 of 10 per-cohort comparisons agreed.
Negative controls. A deliberately trivial benchmark failed the anchors, five broken graders failed them, and with model labels shuffled, random redraws passed in at most 1.4% of simulations. Engineers rated a 5% sample 2.3 for task quality and 2.6 for grader verdicts on a 1-to-3 scale.
Abstract
Teams deploying coding agents need benchmarks built from their own code, but curating and validating them by hand is resource intensive, and an unvalidated benchmark can mislead silently. We present Benchmark Factory, an autonomous meta-agent loop with human checkpoints: an LLM agent writes sourcing queries over landed production code changes, runs models of known relative strength on each candidate, diagnoses why a candidate fails, and repairs it by editing its queries, prompts, and configuration until it clears quality anchors that need no per-task human labels. We used Benchmark Factory to build benchmarks for five developer cohorts at our organization. Engineering raters scored a 5% sample 2.3 for task quality and 2.6 for grader verdicts on a 1-to-3 scale. On configurations never used to build them, the benchmarks ranked Claude Opus~4.7 at high reasoning effort above low effort (60.6% vs. 54.4% of tasks passed, $p=0.002$) and GPT-5.5 above GPT-5.4 (65.8% vs. 60.5%, $p=0.004$); 9 of the 10 per-cohort comparisons pointed the same way. A deliberately trivial benchmark failed the anchors; five broken graders failed them, even under a relaxed rule fixed in advance that the real grader passes; and with model labels shuffled so that no true ordering existed, random redraws of a candidate passed in at most 1.4% of simulations. The anchors are necessary, not sufficient, conditions for a valid benchmark, designed to focus human review on what the loop cannot repair.