SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Pujun Zheng at East China Normal University with Shanghai Artificial Intelligence Laboratory audits SWE-Bench Pro, finds reward hacking through gold-solution leakage and task-quality defects, and releases SWE-Bench Pro Verified, on which several models score substantially lower than previously reported.
Ask this paper
Two independent sources of unreliability: Reward hacking is enabled by leakage of gold solutions or hidden evaluation information; separately, misleading problem statements and improperly scoped tests inflate scores for reasons unrelated to agent skill.
Anti-hacking without breaking the agent: The safeguards close the major leakage channels while leaving normal agent functionality intact, which matters because a benchmark that blocks legitimate repository exploration measures something different again.
Minimal task refinement: Flawed instances are corrected minimally rather than rewritten, so the verified benchmark stays comparable to the original where the original was sound.
Published numbers were too high: Evaluations on the verified version show some models performing substantially worse than reported, which means existing SWE-Bench Pro results overestimate real software engineering capability.
Why it matters now: SWE-Bench Pro has become a standard reference for repository-level agent evaluation, so a leakage channel in it propagates into every comparison built on it.
Abstract
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: \textbf{reward hacking}, enabled by leakage of gold solutions or hidden evaluation information, and \textbf{task quality issues}, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present \textbf{SWE-Bench Pro Verified}, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines \textbf{anti-hacking} safeguards that eliminate major leakage channels without disrupting normal agent functionality, with \textbf{task refinement} that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.