Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
Guojun Zhu and colleagues at the University of Chinese Academy of Sciences and the National University of Singapore introduce CHASE, which prevents automatic harness optimization from producing harnesses whose benchmark gains depend on shortcuts in the benchmark protocol rather than on the tasks.
Ask this paper
The failure mode: A Proposer that edits prompts, memory, retrieval, tools and control code against a released benchmark can find protocol-wide shortcuts that task holdout does not detect, because holdout changes tasks but not the protocol.
Challenger loop: After each Proposer update, a Challenger searches for an executable protocol transformation that destroys the most gain; a validity firewall confirms the task semantics are unchanged before the counterfactual enters an archive.
Guarantees: The paper defines a shortcut-neutralized benchmark and gives statistical guarantees linking a finite counterfactual archive to it.
OfficeQA result: On OfficeQA, CHASE keeps strong released-benchmark gains while losing much less under valid protocol changes; HarnessCompass, with a fixed generalization gate, scored below the initial harness on the Pro V2 release.
Abstract
Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark $B_{\mathrm{rel}}$ to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed target agent. Task holdout varies semantic tasks but leaves the benchmark protocol fixed, so a "bad genius" Proposer can produce a cheating harness whose released-benchmark gain depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over validity-preserving benchmark counterfactuals. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a confirmation set determines whether the counterfactual enters a finite archive. We formalize an exact shortcut-neutralized benchmark $B_0$ and establish statistical guarantees linking finite counterfactual archives to $B_0$ and characterizing sequential Challenger search. We evaluate CHASE on a synthetic benchmark and on OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes.