SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction
Hexuan Deng, Tianwen Jiang, Jihong Zhang and colleagues at Tencent Hy AI Data (with Beijing Zhongguancun Academy) introduce SWE-Journey, a benchmark that tests coding assistants on long development tasks with simulated users of different skill levels.
Ask this paper
Task horizon. A weak-to-strong synthesis pipeline automatically builds long chains of development work in a continuously evolving repository.
Interaction. Four user personas mined from real interaction data drive a user-simulation agent, so requirements arrive over many turns.
Results. Models pass over 75% of tests for requested functionality when working with software architects, but fewer than 25% with non-coders.
Diagnosis. The authors attribute the gap to three capabilities: asking the right clarifying questions, finding the right code and fixing it correctly.
Abstract
Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length. Code assistants require completing long chains of development work in continuously evolving repositories, while repeatedly clarifying requirements and adapting implementations through multi-turn interaction. To address these gaps, we introduce SWE-Journey, a benchmark for more realistic evaluation of coding assistants. To address the task-horizon gap, we propose a weak-to-strong synthesis pipeline that automatically constructs long-horizon coding tasks. To address the interaction gap, we mine four representative user personas from real interaction data and build a user-simulation agent to reproduce realistic code-assistance interactions. On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders. These results show that current coding assistants still fall short of enabling reliable coding for non-coders. We further analyze the reasons for this gap and identify asking right, finding right, and fixing right as key capabilities during interaction.