Veritas: A Verifier Generation Harness for Agent Tasks
Daoan Zhang, Qinbo Bai, Dehai Min and colleagues at ByteDance (with UIC and NYU) build Veritas, a harness that writes verifiers for agent tasks automatically, since self-improving systems are only as good as the verifier that judges each round.
Ask this paper
Mechanisms. Three additions to a general agent harness: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles partly refined by a self-improvement loop.
Evaluation. Six benchmarks: four scored by execution, whose official verifiers the authors certify first, and two pairwise-comparison benchmarks. Agreement is Cohen's kappa against official labels, with a missing verdict counted as an error.
Result. With the model fixed, Veritas verifiers agree with official verdicts better than seven other harnesses (including Claude Code, Codex CLI and Gemini CLI) on all four execution-scored benchmarks, and are at or near the best on the pairwise ones.
Failure mode found. Plain harnesses often fail by not delivering a verdict at all. On Terminal-Bench, pi has the best agreement on judged items but leaves 67 of 189 candidates without a verdict.
Transfer. The mechanisms improve 9 of 12 harness-benchmark pairs across three harnesses, and beat the vendor's own harness for two of three other model families.
Abstract
The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.