Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents
Ruiqing Yue, Yu Cui and colleagues (Chinese Academy of Sciences and Beijing Institute of Technology) speed up self-evolving agent harnesses by separating failures caused by the model from failures caused by the harness, and repairing only the recurring harness-level ones.
Ask this paper
Problem with current harness evolution: Iterative search evaluates and revises candidate harnesses against individual task failures. It is slow because of repeated agent runs and code edits, and it overfits to observed tasks and failure patterns.
Diagnosis first: A single failure can come from a model deficiency or a systematic harness deficiency. Optimizing against each failure produces model-specific accommodations that do not transfer.
Cross-instance aggregation: Ecdysis analyzes failure evidence from a batch of task instances together and biases changes toward patterns that recur across tasks.
Failure-Driven Collaborative Refinement: Multiple diagnostic roles identify failure causes and iteratively refine a specification of the harness change before code is modified.
Results: Up to 1.84x faster harness training than existing harness evolution methods, with 18.56% higher reasoning accuracy for the resulting harnesses.
Abstract
Self-evolving runtime harnesses can substantially improve the capabilities of large language model (LLM) agents and provide a promising paradigm for optimizing agent execution. Existing harness evolution methods typically rely on iterative search, repeatedly evaluating and revising candidate harnesses based on execution feedback from task instances. While this paradigm enables continuous harness optimization, it incurs substantial time overhead due to repeated agent executions and code modifications, and may overfit to observed tasks and specific failure patterns, resulting in degraded generalization to unseen tasks. We identify the lack of principled failure diagnosis as a key bottleneck in harness evolution: an observed failure can reflect either model-specific deficiencies or systematic harness deficiencies, and directly optimizing against individual failures can lead to unnecessary model-specific accommodation. We therefore propose Ecdysis, an efficient and effective framework that distinguishes model-specific accommodation from harness-level repair and biases adaptation toward systematic harness deficiencies by identifying recurring cross-task failure patterns. Ecdysis adopts a batch-level cross-instance failure aggregation paradigm to jointly analyze failure evidence from multiple task instances and further introduces Failure-Driven Collaborative Refinement to diagnose failure causes and iteratively refine harness modification specifications. By combining cross-instance failure analysis with multi-role diagnosis, Ecdysis enables more effective harness evolution with lower training time. Experiments show that Ecdysis achieves up to a 1.84x speedup in harness training compared with existing harness evolution methods, while improving the reasoning accuracy of the resulting harnesses by 18.56%.