HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu and a large multi-institution team introduce HarnessDev, which moves the unit of evaluation from a model's task outputs to the runnable agent harness it can build and then improve, across creation and evolution stages.
Ask this paper
The premise. Agent capability increasingly lives in model-external execution infrastructure, and changing the harness while holding weights fixed substantially changes task performance. Current evaluations report performance under a chosen harness and never test whether the model can build one.
Two stages. In Creation the agent starts from a minimal seed and a few cases and builds a complete execution system. In Evolution it revises its own harness using downstream execution feedback.
Scored on capability and cost. Each constructed harness is evaluated on held-out task success and on execution-token cost, which prevents the trivial strategy of buying accuracy with unbounded spend.
Scale of the study. Six creator LLMs, four domains, five downstream benchmarks, 2,207 unique downstream instances, with hidden evaluation tasks withheld from development.
The results are mixed in an informative way. Generated harnesses lag mature human-engineered references on code and on search and research, but match or exceed them on writing and ML experimentation, with large cost variance. Evolution gains are unstable, transfer only partially to held-out tasks, and depend strongly on which model executes the harness.
Abstract
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.