Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
Mykhailo Kozyrev and colleagues at JetBrains Research test whether automatically optimized SKILL.md files help a coding agent on real repository work, using tasks mined from each repository's own merged pull requests.
Ask this paper
Harder tasks: Merged pull requests are reverted at one frozen base commit and become tasks, because the injected-defect tasks used in prior work are small enough that a capable agent solves them with no document.
Scoring: A candidate SKILL is scored by whether the same agent does better with it than with the seed SKILL on the same task.
Result: On three Kotlin repositories (kotest, ktor, koog), GEPA-optimized SKILLs raise the score by 4.9 points on average and SkillOpt by 0.1 points.
Variance: At the number of tasks one repository's history provides, the 4.9-point gain cannot be separated from the agent's run-to-run variance.
Maintainer review: A koog maintainer found repository knowledge in the SKILLs that is only available from working in the project, and with either SKILL the agent reached its goal faster and at lower cost.
Abstract
Coding agents increasingly read repository knowledge from SKILLs --- plain \texttt{.md} files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a benchmark. A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all. We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by $4.9$pp on average, and the ones SkillOpt finds leave it where it started, $0.1$pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.