Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Wenxuan Ding, Changlong Yu, Sha Li, Greg Durrett and colleagues (Amazon and NYU) introduce Harness-Delta Attribution, a protocol that splits the gain from automatic harness evolution into search-set overfitting, test-time scaling, and a residual upper bound on generalizable improvement.
Ask this paper
Question. Harness evolution optimizes prompts, skills, tools and orchestration code around a fixed model on a search set. The paper asks how much of the reported gain would survive on new tasks.
Scope. HDA is applied to 11 executor models and three harness-evolution frameworks on math, coding, creativity and agentic tasks. 21 settings show statistically significant search-set gains.
Decomposition. Across those 21 settings, overfitting accounts for 54% of the gain on average and extra test-time compute for 14%. Their combined share exceeds 50% in 14 of the 21.
Transfer. Search-set gains often fail to transfer to held-out tasks, and validation-based signals do not prevent the overfitting.
Why it matters. Auto-harness results are now common in agent papers. This gives reviewers and practitioners a concrete check: report how much of the gain remains after removing overfitting and compute effects.
Abstract
Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.