Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents
Yiran Wu (Penn State, partly at Microsoft Research), Qingyun Wu and colleagues propose Behavioral Self-Rewarding (BSR), which adds bounded auxiliary rewards from the policy's own pairwise judgments of its SWE trajectories, while keeping the verifier pass/fail order.
Ask this paper
Problem. Binary test rewards give no contrast within groups where all trajectories pass or all fail.
Method. The policy compares its own trajectories pairwise; preferences become task-local scores mapped into bounded rewards that cannot reverse verifier outcomes. No new environment interactions are needed.
Results. On SWE-bench Verified, Qwen3-14B improves from 32.1% to 34.7% average accuracy at a 400-step budget; on SWE-bench-Live from 6.33% to 8.11%.
Ablations. The full mapping reaches 33.2% average accuracy against 32.0% without its components, and failure-only refinement gives the best Pass@3 at 46.6%.
Scope. Tested on Qwen3-14B and Qwen3-32B configurations.
Abstract
Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We investigate whether the current policy can judge its own trajectories and use these judgments to improve task performance without a separately trained reward model. To this end, we introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare observable debugging behavior among trajectories from the same task with the same verifier outcome. BSR aggregates pairwise preferences into task-local scores and converts them into bounded auxiliary rewards, introducing reward contrast while preserving the verifier-defined pass/fail ordering. This provides additional supervision from already-collected trajectories without requiring new environment interactions. Experiments on SWE-bench Verified and SWE-bench-Live show consistent improvements across the evaluated Qwen3-14B and Qwen3-32B configurations, supporting using the policy's own comparative judgments as a complementary learning signal for training software engineering agents.