Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Yuyao Ge, Shenghua Liu, Xueqi Cheng and colleagues (Institute of Computing Technology CAS, Tsinghua, Peking University) introduce Doc2Harness, a benchmark in which a model must build a complete coding-agent harness from a development document and an interface-only contract.
Ask this paper
Stage 1. A build model inside a fixed builder harness gets a contract skeleton and a document consolidating behavior across twelve frontier harnesses, then builds a candidate harness in an isolated container.
Stage 2. The frozen harness is run per test case and driven end to end by a scripted mock model, so results are deterministic.
Scoring. Four observable channels: requests sent to the model, files on disk, process exit code and command output, with positive controls against vacuous passes.
Results. Across nineteen frontier models, three builds each on 328 test cases, GPT-6 Astra leads at 53.05% and the weakest model reaches 27.44%.
Failure pattern. Failures cluster on how cross-cutting functionality is split across modules.
Abstract
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.