Rethinking Verification in Agent Harnesses
Zhengxin Zhang, Erkang Zhu, Yin Lin, Claire Cardie, Bolin Ding and Jingren Zhou (Alibaba Group, Cornell University) study verification as an axis of test-time scaling in agent harnesses, comparing 17 designs that differ in when the harness intervenes, who checks the work and how checks lead to revision.
Ask this paper
Four layers. The framework covers initial instructions, mid-turn reminders, completion gates and independent review, evaluated across seven frontier coding models on Terminal-Bench and SWE-bench Verified, plus a new VisualBench for expert evaluation of visual and interactive artifacts.
Instructions and reminders. Initial instructions and execution-time reminders give modest, model-dependent average gains without substantially increasing output tokens. Completion gates can give larger gains but may substantially increase cost.
Independent review. A separate critic improves average accuracy by 5.9 to 9.5 points on Terminal-Bench and 3.9 to 6.6 on SWE-bench Verified at 1.13 to 1.25x and 1.00 to 1.04x baseline cost. On VisualBench it reaches net preferences of 13.2 to 23.8 points over the baseline.
More review. Additional review rounds or additional independent critics raise average Terminal-Bench accuracy by up to 3.5 more points.
Abstract
Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification as an axis of test-time scaling by examining when to intervene, who checks the work, and how checking leads to revision. Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review. We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), and introduce VisualBench for expert evaluation of visual and interactive artifacts. Initial instructions and execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, without substantially increasing average output token use. Completion gates can yield larger gains but may substantially increase cost. Independent review by a critic offers a more favorable accuracy-cost tradeoff, improving average accuracy by 5.9-9.5 percentage points on TB and 3.9-6.6 on SWE at 1.13-1.25 and 1.00-1.04 times baseline cost, respectively. On VisualBench, independent review achieves net preferences of 13.2-23.8 points over the baseline across its four designs, exceeding the evaluated L0-L2 designs. Additional review rounds or independent critics further improve average TB accuracy by up to 3.5 percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.