VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
Zekai Wang and Ashia C. Wilson (MIT) with Yingqiang Ge, Chandan K. Reddy and colleagues at Amazon introduce VERSE, a harness optimizer that also improves its own prompts, skills and tools, and that tests its draft edits by execution before submitting them.
Ask this paper
Verification is required. In a controlled study, letting the optimizer evolve itself does not help without execution-based checks, and gives the best result in that study when checks are available.
Method. VERSE lets the optimizer test draft edits, replay failures and perturb suspected steps before submission, and tracks fixes and regressions across rounds. Both the executor harness and the optimizer's own harness are revised; model weights stay fixed.
Theory. The authors prove that when recorded evidence is ambiguous, reading trajectories alone leaves a minimum rate of wrong repairs, which executed checks can lower.
Results. VERSE improves all four published harness optimizers it is added to on held-out SWE-rebench tasks and on newer out-of-distribution tasks in five languages. The best harness reaches 42.3% and 37.7% against 39.2% and 29.3% for the strongest baselines.
Abstract
Harness evolution improves an LLM agent's prompts, tools, and workflow, while the optimizer's own tools and procedures often remain fixed. We study whether an optimizer can improve another agent more effectively by also improving how it diagnoses failures, develops edits, and tests their effects. Two observations guide our design. In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available. Across five executors, self-evolving optimizers build their own tools for failure analysis, verification, training audits, and workflow control. Motivated by these findings, we introduce VERSE, a Verified Self-Evolving optimizer for agent harnesses. VERSE lets the optimizer test draft edits, replay failures, and perturb suspected steps before submission, while tracking fixes and regressions across rounds. Using this feedback, the optimizer revises both the executor harness and its own prompts, skills, tools, hooks, and notes, while the weights of the optimizer and executor models stay fixed. Under a shared protocol with disjoint training, validation, and test tasks, VERSE improves all four evaluated harness optimizers on held-out SWE-rebench tasks and newer out-of-distribution tasks in five languages. Its best validation-selected harness reaches 42.3% and 37.7% accuracy, respectively, against 39.2% and 29.3% for the strongest baselines. Code is available at https://github.com/wzekai/VERSE.