Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
Mohsen Arjmandi (evolutionID GmbH) tests whether a vendor's own agent harness solves more coding tasks than a neutral harness on the same model, using paired runs on a private, contamination-controlled suite.
Ask this paper
Design: The same 80 tasks run under claude-agent-sdk and deepagents on Opus 4.8, and under the openai-codex SDK and deepagents on GPT-5.5, with 792 of 800 planned runs graded by an isolated oracle.
No average advantage: The native harness differs by -1.25 points on Opus 4.8 (48.8% vs 50.0%, 95% CI -10.0 to +7.5) and +1.25 points on GPT-5.5, so neither contrast resolves a winner.
Opposite strata: On Opus the native harness trails by 9.0 points on 61 repository tasks and leads by 23.7 points on 19 contest tasks (p = 0.003), a split chosen after seeing the data that needs replication.
Completion versus correctness: 22 of 81 runs cancelled at the wall-clock ceiling had already produced a passing patch.
Cost accounting defect: Input-token fields are cache-inclusive on some SDKs and cache-exclusive on others, and this error invalidated the cost figures in the August manuscript; the corrected neutral harness costs 1.3 to 1.6 times as much per solved task on Opus 4.8.
Abstract
An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. We measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003). The partition was chosen after seeing the data and needs a designed replication. Correctness and completion also separate: 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5. These are observed-usage estimates. On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved. This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1). We release the orchestrator, grading oracle, reanalysis code and derived aggregates. The tasks stay private.