CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering
Prince Zizhuang Wang (CMU) and colleagues at USC, UW-Madison and other universities introduce CUA-SWE, a benchmark and environment where agents must both edit code and operate the running application through its GUI to diagnose, fix and verify bugs across web, game, mobile and DevOps projects.
Ask this paper
Setup. Each task ships an editable project plus the running software. Agents decide when to read code, run commands, click through the app and inspect screenshots, and deterministic tests check the final behavior.
Two questions. How agents use GUI feedback to diagnose and verify repairs, and whether they can finish tasks when part of the specification is only visible in the running application.
Results. GPT-6-Astra leads the four-domain comparison at 59.9% mean task success, 17.7 points above GPT-5.6-Sol. In a three-attempt game study it solves 34.5% of tasks on all three attempts against 3.4% for GPT-5.6-Sol.
Training signal. Supervised fine-tuning on the environment's trajectories raises a smaller model's native-verifier pass rate from 11.1% to 33.3%.
Abstract
Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored. Diagnosing a runtime interaction failure requires agents to connect visual observations with the responsible code, then use the application again to verify the repair. We introduce CUA-SWE, a benchmark, environment, and evaluation pipeline for software engineering with computer use. Beyond studying how GUI feedback supports diagnosis and repair, we ask whether agents can complete software engineering tasks when required specification or operational information is available only through the running application's visual interface. CUA-SWE spans four software engineering domains and requires agents to modify code and configuration, execute commands, interact with running software, and inspect visual feedback within the same task. Each task includes deterministic, task-specific tests that verify whether the resulting software satisfies the requirements and preserves specified behavior. Our evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes. We examine performance across domains and task information requirements, alongside the development behaviors associated with successful repairs. CUA-SWE provides a unified testbed for studying how agents use visual feedback and interaction to guide software engineering, with executable correctness criteria for the resulting software.