ProductWebBench: Evaluating Website Continuity Beyond Task Completion
Jiajun Wu, Jian Yang and colleagues (Beihang University, IQuest Research, Huawei) introduce ProductWebBench, which grades frontend coding agents both on the requested change and on whether the rest of the site keeps working.
Ask this paper
Setup. 320 Change tasks on existing repositories and 80 long-horizon Build tasks. Each freezes a repo snapshot, staged requirements, browser evidence and deterministic checks; a simulated user reveals requirements stage by stage and earlier checks stay armed.
Main result. Across 8,737 runs from 13 models, 28.2% of Change runs that pass every requested-work check still fail a continuity check such as a blank route, horizontal overflow or console errors.
Regressions. 10.8% of runs push a browser state they had already passed back to failing. Because checks also run on the untouched repo, each regression has exact provenance: 18.3% of 586 comparable runs break a verified state.
Interaction gap. 29.5% of runs whose static DOM passes still fail on interaction.
Abstract
Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation website continuity, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, 28.2% still fail a continuity check -- a blank route, horizontal overflow, console errors -- and 10.8% push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, 18.3% are caught breaking a verified state. Interaction is a further blind spot, failing in 29.5% of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.