Mining Agent Skills from Production Traces
Yue Ran Kang and Colton Mikolajczyk (MIT) with Chhaya Methani, Hazel Mak and colleagues at Microsoft run a controlled study of how agent skills mined from execution traces depend on what evidence the miner sees and what form the skill takes.
Ask this paper
Design. The mining pipeline is fixed and six configurations are compared. Evidence is successes only, successes and failures with labels ("Goldilocks"), or the same mix with labels withheld. Skills are either an ordered workflow plan or a declarative ontology of entities, states and policies.
ThinkingBox-Bench. Workflow plans beat ontologies by 1.7 pp, labeled mixed evidence beats success-only by 2.4 pp, and withholding outcome labels costs 3.1 pp. Withheld labels simulate production, where whether a run succeeded is often unknown.
APEX-Agents. Ontologies are moderately preferred and evidence regime makes no clear difference, so no single configuration wins across domains.
Task-level variation. Within domains the differences are large. In consulting, workflow plans lead ontologies by 12.1 pp on client-system access and trail by 12.9 pp on document access. In retail, outcome labels help workflow plans by 28.0 pp on order and account changes.
Measurement care. Results are paired per task with task-level bootstrap intervals, and the authors show that a hosted-model control run on a different date drifted from 33.8% to 25.8% pass rate, so controls must be rerun alongside treatments.
Abstract
Agent skills that record procedural instructions are increasingly mined from execution traces rather than curated by hand. Skill-mining pipelines often use known task outcomes or feedback to guide skill construction. In production, reliable information on whether a run has succeeded may be unavailable. We study how the sampling of execution traces, access to success or failure information, and the form of the mined skills affect downstream task performance. Holding the mining pipeline fixed, we compare six combinations of mining evidence and skill forms. Mining evidence has three levels: successful trajectories only, successes and failures with their outcome labels, or the same mix with labels withheld. Skill form has two types: an ordered workflow plan, or a declarative ontology of entities, states, and policies. We evaluate the mined skills on two enterprise benchmarks, ThinkingBox-Bench and APEX-Agents. Analysis of task-level paired differences shows that the benefits of different configurations of mining evidence and skill forms depend on the enterprise domain. On ThinkingBox-Bench, paired differences show that workflows score better than ontology by 1.7 pp, Goldilocks beats success-only evidence type by 2.4 pp and Goldilocks blind simulating skills learnt without outcomes is worse by 3.1 pp. APEX-Agents shows a moderate preference for ontologies and no clear preference between evidence regimes. Within each domain, task structure related constraints drive uneven performance with mined skills. These findings motivate tailoring meta-skills to the demands of the target tasks rather than adopting a one-size-fits-all approach.