Same Model, Different Harness: Different Coding-Agent Results
Sydney Lewis holds the model and task fixed and varies only the harness, showing that a coding agent's benchmark number is a property of the model-plus-harness pair rather than of the weights.
Ask this paper
Single controlled variable: Control supplies the full conversation in time order; treatment keeps the same record but mechanically shortens older tool results as context fills and reacts to repeated or stalled work. Nothing else changes.
Large effect under context pressure: On a 169-task SWE-bench Verified cohort with a 20,480-token window and a fixed 480-second endpoint, mean per-task fail-to-pass fraction rises from 28% to 49% and complete solutions from 43 to 72.
Transfers without retuning: The same frozen treatment lifts both endpoints for three additional models of different designs, so this is not prompt overfitting to one backbone.
Effect collapses with a wide window: On wide-window Qwen3.6 comparisons the arms are close on Verified and Pro; the harness earns its keep specifically under tight context.
Why it matters: The paper's conclusion is an evaluation-policy claim: coding-agent leaderboards should report the harness as part of the tested solver, otherwise model rankings are partly measuring scaffolding.
Abstract
A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.