Beneath the Surface: What We Overlook When We Talk About Agent Evolution
Xiaobo Wang, Zimu Wang, Siyu Ye, Jiaqi Li and colleagues (USTC, UC Berkeley, BIGAI, Shanghai Jiao Tong University, Simple Agent Lab) test whether harness-evolution methods that raise benchmark scores also produce reliable, efficient and transferable improvement.
Ask this paper
Coverage churn. All sixteen evolved agents from four methods solve new tasks while losing tasks they previously solved. In one general-task case, reliability rises 5.2 points while ten previously covered tasks are lost.
Cost. Most reliability gains also increase deployment cost.
Selection. The highest-scoring evolved agent does not produce the largest mean improvement in any of the four pools, and stricter admission rules can discard useful alternatives.
Edits. Specific changes usually outperform generic reminders, but their benefits change when they are combined.
Transfer and EvoAudit. Coverage usually falls when evolved harnesses move to the general-task target. EvoAudit revises the select, analyze, mutate and gate steps using these diagnostics and achieves the best performance.
Abstract
Agent harness evolution enables recursive self-improvement (RSI) without updating model weights. We examine whether higher benchmark scores yield reliable, efficient, and transferable improvement. We evaluate four methods across harnesses, general and coding tasks. Repeated evaluations, controlled agent expansion, component interventions, and admission replays reveal recurring patterns. All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones. In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks. Most reliability gains also increase deployment cost. Specific changes usually outperform generic reminders, but their benefits can change when combined. The highest-scoring agent does not yield the largest mean improvement in any of our four pools, and stricter admission can discard useful alternatives. Transfer selectively preserves gains: coverage usually falls on the general-task target across both harness types and paradigms, but rises on the coding target with the outer-layer-only harness. These findings motivate preservation-aware evaluation, tested decision rules, and explicit applicability conditions. As a constructive follow-up, EvoAudit turns these diagnostics into revisions of select, analyze, mutate, and gate, achieving the best performance.