Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
Cheng Qian, Kunlun Zhu, Beibin Li, Zhenhailong Wang and Heng Ji at Apodex and UIUC study test-time AI-for-AI, where a Builder model with frozen weights learns to construct better execution harnesses for a Target model with frozen weights.
Ask this paper
Meta-skills. The Builder learns principles stating when the Target needs support and what resources to provide, extracted from the Target's execution feedback on a development set.
Frozen skill bank. At test time the Builder uses the fixed bank to build harnesses for unseen tasks.
Results. On Harness-Bench and NewtonBench the full bank improves macro-average performance by 8.95 points over building without skills, and by 12.02 points over handing the same bank directly to the Target.
Self-improvement path. Gains also appear when one model plays both Builder and Target.
Abstract
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.