Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs
Jiacheng Xu and colleagues at Nanyang Technological University with Skywork AI frame automatic test-case generation as an adversarial RL problem, where the generator must produce counterexamples targeted at the solver's current failure modes.
Ask this paper
Why it is adversarial rather than supervised: effective test cases depend on what the solver currently gets wrong, so the target moves as the solver improves and a fixed test corpus goes stale.
Two stages over a rolling policy-aligned buffer: Stage 1 trains the generator to produce tests consistent with the reference solution, establishing soundness; Stage 2 restricts the buffer to current failure modes and learns counterexamples, establishing discrimination.
Both properties are required: high-quality tests must be sound and discriminative, and prior work usually optimizes only one.
Results on TACO and LiveCodeBench: TCS improves both pass@1 and inference-time answer selection using the generated tests.
Transfers beyond its own policy: the learned test generator also selects effectively among outputs from other LLMs, which makes it reusable as a verifier.
Abstract
Reinforcement learning (RL) has substantially advanced code generation with large language models (LLMs) through executable feedback. The feedback for coding problems mainly comes from specific test cases, where high-quality test cases are often scarce since they should be both sound and discriminative. We thus turn to study the auto-generation of test cases using the learned model. We find this is naturally an adversarial RL problem: the model is expected to generate effective test cases as counterexamples, depending on the solver's current failure modes. We propose Test Cases Scaling (TCS), a two-stage RL framework for effective test generation. Both stages train a test generator from a rolling policy-aligned buffer: Stage 1 generates tests consistent with the reference solution, and Stage 2 restricts the buffer to current failure modes and learns counterexample tests. Across TACO and LiveCodeBench, TCS improves both pass@1 and inference-time answer selection according to generated tests. We find the learned test generator also enables effective selection among other LLM outputs.