LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
Yi Wang and colleagues (AMAP / Alibaba with collaborators) benchmark the outer loop rather than the coding agent, evaluating a Controller model that instructs a fixed Worker coding agent after each round and decides when to stop.
Ask this paper
The confound it removes: An end-to-end run cannot tell you whether success came from the loop guidance or the coding agent, so LoopArena fixes the Worker and varies only the Controller.
Three evaluation types: Type I scores next-step Loop Contract selection through execution-validated questions with no Worker run, Type II executes repeated control on a slice of a task, and Type III runs the paired full task from its original state.
Headline number: Best observed Strict Success Rate on full tasks is 24.69%, which says outer-loop control is nowhere near solved even where the inner agent is capable.
Cost result: Across Controllers the paired reduction in estimated inference cost averages 64.4%, and the cheap Type II proxy orders Controllers almost identically to full tasks (Spearman rho 0.9747).
Named failure modes: Trusting a stale progress note, skipping needed verification, spending budget in the wrong direction, and stopping before the task is safe to submit.
Abstract
Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=\textbf{0.9747}\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .