ExecCritic: Learn to Test, Test to Improve for Coding Agents
Leitian Tao (UW-Madison, internship at Microsoft Research) with Baolin Peng, Hao Cheng, Wenlin Yao and colleagues at Microsoft Research present ExecCritic, which separates test writing from source repair into two agents and trains each with its own RL objective, showing that test quality decides whether execution feedback helps at all.
Ask this paper
Bad tests make feedback actively harmful: Holding the base Repair agent fixed, tests from the base Test agent drop the SWE-bench Verified resolved rate from a no-test baseline of 61.2% to 57.3%, while tests from GPT-5.6-sol raise it to 65.3%. Execution feedback is not free.
Role separation prevents correlated errors: When one trajectory writes both patch and test, their mistakes can agree and produce false confidence. ExecCritic has a Test agent generate repository-native tests, a fail-closed harness qualify and freeze them, and a Repair agent revise source code without touching them.
Two training phases: Learn to Test trains the Test agent to produce behaviorally valid tests that separate correct from incorrect patches; Test to Improve trains the Repair agent on both direct resolution and feedback-guided revision.
Post-training moves the Test agent the most: Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%.
Composed result: Both post-trained Qwen-3.5-35B-A3B agents together reach 72.6%, an 11.4-point gain over the no-test baseline without a stronger model or oracle feedback at evaluation time. Code is at github.com/MSR-Orchard/execcritic.
Abstract
Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.