UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users
Zhan-Yu Wu and colleagues at Tencent Hunyuan introduce UserSWE, a benchmark that converts single-turn SWE tasks into multi-turn bug-fixing conversations with simulated developers and runs tests after every turn.
Ask this paper
Simulated users. Information pools limit what the simulated user can reveal at each turn, and a persona chain sampled from a transition model fit on 20,859 real development sessions controls how the user speaks.
Verdicts. Tests run after every turn. Two agents from different model families write and gate each verdict, using test results as primary evidence and model judgment only for what tests cannot decide.
Rankings depend on turn budget. Across eleven agents on 72 tasks from 42 repositories, Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by turn eight, and Claude Opus 5 leads at most budgets in between.
Implication. Single-turn scores do not characterize interactive coding ability, because the order of agents changes with how many turns the user gives them.
Abstract
As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These results indicate that single-turn performance is insufficient to characterize interactive coding ability.